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Job Analysis of the Knowledge and Abilities 
Important for Newly Licensed (Certified) Spanish Teachers 

Richard J. Tannenbaum 

Executive Summary 

A job analysis was conducted to define a knowledge and ability domain important for 
newly licensed (certified) Spanish teachers to perform their jobs in a competent manner. The 
results of the job analysis will be used to develop test specifications for the Spanish subject 
assessments of the Praxis Series: Professionals Assessments for Beginning Teachers ™. 

An initial draft domain of knowledge and ability statements was constructed by 
Educational Testing Service (ETS) Test Development staff with subject-matter expertise in 
Spanish and ETS Research staff with expertise in job analysis. In the process of developing the 
draft domain, ETS subject-matter experts reviewed previous National Teacher Examination 
(NTE) Spanish test specifications, state licensure (certification) requirements, and relevant' 
professional literature. The resultant draft domain consisted of eight major content areas and 
149 knowledge and ability statements. The eight major content areas were: (1) Listening 
Comprehension, (2) Reading Comprehension, (3) Grammatical Structure and Written 
Expressions, (4) Language Analysis, (5) Speaking, (6) Writing, (7) Cultural Background, and 
(8) Pedagogy Specific to Spanish. 

This draft domain was then reviewed by an External Review Panel of 10 subject-matter 
experts: four secondary school Spanish teachers, five teacher educators, and a district 
administrator. The panel reviewed the draft domain for (1) the appropriateness of its overall 
structure and (2) the appropriateness of the specific knowledge and ability statements and their 
completeness and clarity. Revisions suggested by the panel, including additions and deletions of 
content areas and knowledge and ability statements, were obtained via a telephone interview 
conducted by ETS Research staff. Minor wording changes were made to the draft domain and 
some additional statements were included. The revised domain consisted of 154 knowledge and 
ability statements. 

This revised draft domain was then reviewed by an Advisory/Test Development 
Committee. Similar to the External Review Panel, this committee was comprised of secondary 
school Spanish teachers (n=5), teacher educators (n=3), and state administrators (n=2). This 
committee also had representation by race/ethnicity, sex, and geographic region. The purpose 
of this committee was to modify the revised draft domain so that it accurately reflected what the 
members of the committee believed were the knowledge and abilities important for newly 
licensed (certified) Spanish teachers. This modification process occurred during a 4-day meeting 
held at ETS. The major modifications to the domain included: (1) changing Grammatical 
Structure and Written Expressions to Structure of the Spanish Language and incorporating 
Language Analysis into this new content area and (2) changing Pedagogy Specific to Spanish to 
Foreign Language Pedagogy to reflect the generalizability of pedagogy across all foreign 
languages. Other modifications included rewording some knowledge and ability statements and 
adding and deleting some knowledge and ability statements. The modified domain consisted of 
seven major content areas and 166 knowledge and ability statements. The seven major content 
areas were: (1) Listening Comprehension, (2) Reading Comprehension, (3) Structure of the 
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Spanish Language (4) Speaking, (5) Writing, (6) Culture (changed from Cultural Background), 
and (7) Foreign Language Pedagogy. 

This revised domain was then subject to verification/refutation through a national survey 
of several hundred Spanish teachers, state administrators, and teacher educators. The survey 
participants were asked to rate the knowledge and ability statements, using a 5-point scale, in 
terms of importance for newly licensed (certified) Spanish teachers to perform their jobs in a 
competent manner. The 5-point scale ranged from a low of 0 (of no importance) to a high of 4 
(very important). The purpose of the survey administration was to identify a core of knowledge 
and ability statements that relatively large numbers of educational professionals verified to be 
important to newly licensed (certified) Spanish teachers. The latter objective is accomplished by 
analyzing the mean importance ratings provided by the three groups of educational professionals 
(teachers, state administrators, teacher educators) and by appropriate subgroups of respondents 
(sex, race/ethnicity, geographic region, teaching experience). Knowledge and ability statements 
that are judged to be important by all respondent groups and subgroups define the core. The 
core becomes the primary database for the development of test specifications. The derivation of 
test specifications from those knowledge and abilities verified to be important by the surveyed 
educational professionals provides a substantial evidential basis for the content validity 
(relevance) of the Spanish subject assessments. 

Two types of data analysis were conducted to support the development of content valid 
(relevant) test specifications for the Spanish subject assessments: (1) means were computed of 
the importance ratings for each knowledge and ability statement by the three groups of 
educational professionals and by the appropriate subgroups of respondents; and (2) correlations 
of the profiles of these mean importance ratings were computed across the three groups of 
educational professionals and within the appropriate subgroups of respondents. 

A cutpoint of a mean importance rating of 2.50 (the midpoint between moderately 
important (scale value 2) and important (scale value 3)) was established to identify the core of 
important knowledge and ability statements. Statements that were judged, by all three 
respondent groups and all subgroups of respondents, to be 2.50 or higher comprised the core, 
and therefore, were considered eligible for inclusion in the development of test specifications. 
(However, because the survey participants were not involved in the development of the 
knowledge and ability domain, they may lack certain insights that the Advisory /Test 
Development Committee members have due to their high level of involvement in the definition 
of the domain. As a consequence, if the committee believes that a knowledge or ability 
statement rated below 2.50 should be included in the specifications and the cor/imittee can 
provide compelling written rationales, those knowiedge and ability statements may be reinstated 
for inclusion in the test specifications.) 

The results of the mean analysis conducted by teachers, state administrators, and teacher 
educators revealed 15 knowledge and ability statements that were rated less than 2.50. This 
represents only 9% of the content domain. Six additional knowledge and ability statements were 
rated below 2.50 by the subgroup of teachers with five or less years of experience. In total, only 
21 of the 166 statements (13%) did not meet the 2.50 criterion for inclusion. 

The computation of correlation coefficients to assess agreement in terms of perceived 
relative importance of the knowledge and abJity statements revealed a very high level of 
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agreement. The coefficients for comparisons among teachers, state administrators, and teacher 
educators all exceeded .85; and the coefficients generated by the subgroup analyses all exceeded 
.90. These findings, coupled with the outcomes of the mean analyses indicate that there is 
substantial agreement in the importance ratings across the diversity of educational 
professionals. 

The 145 knowledge and ability statements that were verified to be important by the 
surveyed teachers, state administrators, teacher educators, and the subgroups should be used 
as the foundation for the development of test specifications. Test specifications that are linked 
to the results of a job analysis provide support for the content validity of the derived assessment 
measures and may be considered as part of an initial step in ensuring the fairness (to subgroups 
of Spanish eacher candidates) of the derived assessment measures. It is reasonable to assume 
that, due to testing and psychometric constraints (e.g., time limits, ability to measure reliably 
some content), not all of the verified content may be included on the assessment measures. One 
source of information that may be used to guide the Test Development Committee in their 
decision of what verified content to include on the assessment measures is the mean importance 
rating. Although a rank ordering of the content by mean importance rating is not implied, it is 
recommended that initial consideration be given to content that is well above the cutpoint and 
represents the appropriate breadth of content coverage. 

Evidence was also provided in this study of the comprehensiveness of the content 
domain and the judged importance of the seven major content areas. These two pieces of 
information have implications for the adequacy of the content domain. If the domain was 
adequately defined then the knowledge and ability statements should be judged to have been 
well covered by each major content area and each major content area should be judged to be 
important. The results support the adequacy of the defined content domain. The knowledge 
and ability statements were judged to have been well covered (a value of 4.0 on the content 
coverage scale); and the major content areas were judged to be important (a value of 3.0 on the 
importance scale). 



A Job Analysis of the Knowledge and Abilities Important 
for Newly Licensed (Certified) Spanish Teachers 

Introduction 

The foreign language subject assessments of The Praxis Series: Professional Assessments 
for Beginning Teachers™ are designed to assess a prospective foreign language teacher's content 
knowledge of a specific subject area, general pedagogical knowledge, subject-specific pedagogical 
knowledge, and ability to communicate in written and verbal forms. The series will be used by 
various states as part of their teacher licensure (certification) process. One ot these foreign 
language assessments covers the subject area of Spanish. To identify the content domain of this 
examination and to support the content validity (content relevance) of this examination, a job 
analysis was conducted of the knowledge and abilities important for newly licensed (beginning) 
Spanish teachers. This report will describe the job analysis. In particular, it will present the 
(1) methods used to identify and define the job-related knowledge and abilities, (2) types of 
statistical analysis conducted, (3) results of these analyses, and (4) implications of the results for 
developing test specifications. 

Standards for Educational and Psychological Testing 

The Standards for Educational and Psychological Testing (1985) is a comprehensive 
technical guide that provides criteria for the evaluation of tests, testing practices, and the effects 
of test use. It was developed jointly by the American Psychological Association (APA), the 
American Educational Research Association (AERA), and the National Council on 
Measurement in Education (NCME). The guidelines presented in the Standards have, by 
professional consersus, come to define the necessary components of quality testing. As a 
consequence, a test.ng program that adheres to the Standards is more likely to be judged to be 
valid (defensible) than one that does not. 

There are two categories of criteria within the Standards, primary and secondary. Those 
classified as primary "should be met by all tests . . . unless a sound professional reason is 
available to show why it is not necessary, or technically feasible, to do so in a particular case. 
Test developers and users . . . are expected to be able to explain why any primary standards 
have not been met" (AERA/APA/NCME, 1985, p. 2). One of the primary standards is that the 
content domain of a licensure or certification test should be defined in terms of the importance 
of the content for competent performance in an occupation. "Job analyses provide the primary 
basis for defining the content domain." (P. 64). 

The use of job analysis to define the content domain is a critical component in 
establishing the content validity of licensure and certification examinations. Content validity is 
the principle validation strategy used for these examinations. It refers to the extent to which the 
content covered by an examination overlaps with the important components (tasks, knowledge, 
skills, or abilities) of a job (Arvey & Faley, 1988). Demonstration of content validity is 
accomplished through the judgments of subject-matter experts. It is enhanced by the inclusion 
of large numbers of subject-matter experts who represent the diversity of the relevant areas of 
expertise (Ghiselli, Campbell, & Zedeck, 1981). The lack of a well-designed job analysis is 
frequently cited (by the courts) as a major cause of test invalidity. 
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Job Analysis 



Job analysis refers to procedures designed to obtain descriptive information about the 
tasks performed on a job and/or the knowledge, skills, and abilities thought necessary to 
perform adequately those tasks (Gael, 1983). The specific type of job information collected by a 
job analysis is determined by the purpose for which the informat'on will be used. Foi purposes 
of developing licensure and certification examinations, a job analysis should identify the 
important knowledge or abilities necessary to protect the public -- interpreted as the impoi tance 
of the content for competent performance in an occupation (Standards for Educational and 
Psychological Testing, AERA/APA/NCME, 1985). In addition, a well-designed job analysis 
should include the participation of various subject-matter experts (Mehrens, 1987); and the data 
collected should be representative of the diversity within the job. Diversity refers to regional or 
job context factors and to subject-matter-expert factors such as race\ethnicity, experience, and 
sex (Kuehn, Stallings, & Holland, 1990). The job analysis conducted for Spanish was designed to 
be consistent with the Standards and current professional practices. 

Objectives of the Job Analysis Study 

The objectives of this study were: (1) to construct a comprehensive domain of 
knowledge and abilities that is important for newly licensed (certified) Spanish teachers; and 
then (2) to obtain, using survey methodology, the independent judgments of a national sample of 
Spanish educational professionals (teachers, teacher educators, state administrators) to verify or 
refute the importance of the domain of knowledge and abilities. The verification/refutation 
component serves a critical role to ensure that the domain (in whole or in part) is judged to be 
relevant to the job of a newly licensed (certified) Spanish teacher by a diversity of educational 
professionals. It is those knowledge and abilities that are verified to be important that will be 
used in the development of test specifications for the Spanish subject assessments of The Praxis 
Series: Professional Assessments for Beginning Teachers ™. 

Method 

In overview, the methodology consisted of defining the knowledge and abilities important 
for newly licensed (certified) Spanish teachers to perform their jobs in a competent manner. 
This was accomplished first by having subject-matter experts define a domain of knowledge and 
abilities important for newly licensed (certified) Spanish teachers and then by presenting these 
judgments for verification or refutation through a national survey of Spanish educational 
professionals. This functions as a "check and balance" on the judgments of the subject-matter 
experts and reduces the likelihood that unimportant knowledge and abilities will be included in 
the development of the test specifications. The survey participants were Spanish teachers, state 
administrators, and teacher educators whose names were obtained from the memberships of the 
(1) American Association of Teachers of Spanish and Portuguese, (2) American Council on the 
Teaching of Foreign Languages, and (3) National Council of State Supervisors of Foreign 
Languages. The participants were asked to rate knowledge and ability statements in terms of 
importance foi newly licensed (certified) Spanish teachers to perform their job in a competent 
manner. The specific steps in the job analysis process are described below. 
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Build a Draft Domain of Knowledge and Abilities 



The first step in the process of conducting the job analysis was to construct a preliminary 
knowledge and ability domain. This draft would function as the initial definition of the 
knowledge and ability domain of newly licensed (certified) Spanish teachers. The domain was 
constructed by Educational Testing Service (ETS) Test Development staff with subject-matter 
expertise in Spanish and ETS Research staff with expertise in job analysis. In the process of 
developing the draft, the ETS subject-matter experts reviewed previous National Teacher 
Examination (NTE) Spanish test specifications, state licensure (certification) requirements, and 
relevant professional literature. 

The draft domain for Spanish consisted of eight major content areas and 149 specific 
knowledge and ability statements. The eight major knowledge areas were: (1) Listening 
Comprehension: The ability to understand spoken Spanish; (2) Reading Comprehension: The 
ability to understand standard written Spanish; (3) Grammatical Structure and Written 
Expressions: Know and recognize correct grammatical structure and appropriate style in standard 
written Spanish; (4) Language Analysis: Knowledge of the structural components of the Spanish 
language; (5) Speaking: The ability to speak standard Spanish; (6) Writing: The ability to write 
standard Spanish; (7) Cultural Background: Knowledge of the culture, society, and geography of 
Spanish-speaking countries; and (8) Pedagogy Specific to Spanish: In addition to content, Spanish 
teachers also need to know something about pedagogy specific to Spanish. This might include 
knowledge of students, curriculum, teaching strategies, and resources. 

Review of Draft Domain by External Review Panel 

After the draft domain was constructed it was mailed to a panel of 10 external subject- 
matter experts. These subject-matter experts were secondary school teachers (n = 4), teacher 
educators (n = 5) and a district administrator responsible for overseeing foreign language 
curriculum. The panel also had representation by race/ethnicity, sex, and geographic region 
(see Appendix A for list of members). The purpose of the External Review Panel was to review 
the draft domain in terms of: (1) the appropriateness of the overall structure (i.e., do the major 
content areas adequately define the important components of the knowledge and ability domain) 
and (2) the appropriateness of the specific knowledge and ability statements and their 
completeness and clarity. In addition, the members of the panel were asked to identify other 
content areas and/or knowledge and ability statements that they believed should be added to the 
domain. The recommendations of the panel were obtained via telephone interviews conducted 
by ETS Research staff. ETS Test Development staff then reviewed the compiled 
recommendations of the panel. Based upon the revisions suggested by the panel and Test 
Development staffs re-evaluation of the domain, some modification occurred. 

The eight major content areas of the domain remained the same. However, several of 
the knowledge and ability statements were reworded. Some statements were deleted due to 
their overlap with other statements or perceived lack of relevance to a beginning (newly 
licensed) Spanish teacher. Where perceived necessary, additional knowledge and ability 
statements were included in the domain. The revised domain consisted of approximately 154 
nowledge and ability statements. 
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Advisory/Test Development Committee Meeting 



Consistent with a content validity framework, the job analysis study was designed to 
obtain input from many subject-matter experts at several critical points in the domain definition 
process. To this end, an Advisory/Test Development Committee of secondary school teachers 
(n = 5), teacher educators (n = 3), and state administrators (n = 2) with expertise in Spanish was 
formed (see Appendix B for list of members). This committee also had representation by 
race/ethnicity, sex, and geographic region. The purpose of this committee was to review the 
draft domain (revised by the External Review Panel) in terms of its overall structure, 
completeness, appropriateness of the knowledge and ability statements, and clarity of wording. 
In addition, the members were asked to identify other content areas and/or knowledge and 
ability statements that they believed should be added to the domain and to delete knowledge 
and ability statements that they believed should not be included in the domain. In essence, the 
members were asked to modify the domain so that it accurately reflected what the committee 
believed were the knowledge and ability statements important for newly licensed (certified) 
Spanish teachers. The committee also reviewed and approved the rating scale for the national 
survey and the biographical data that would be asked of the survey participants. The 
biographical data were collected to describe the composition of those who returned completed 
surveys and to permit analysis of the survey responses by various subgroups of respondents (e.g., 
males and females). 

The revision process occurred during a 4-day meeting held at Educational Testing 
Service. The meeting was led jointly by ETS Test Development and Research staff. (Prior to 
the meeting, the members of the committee were mailed a copy of the draft domain to review. 
They were informed about the purpose of the meeting and asked to come prepared to discuss 
their review of the draft domain.) During the course of the meeting, the domain was revised to 
reflect the consensus of the committee. 

One content area, Grammatical Structure and Written Expressions was changed to 
Structure of the Spanish Language: Demonstrate the correct formation and use of grammatical 
structure and appropriate style in written and spoken Spanish. This change also resulted in the 
inclusion of the content area, Language Analysis. The content area, Pedagogy Specific to 
Spanish was changed to Foreign Language Pedagogy (this was done to reflect the perceived 
generalizability of pedagogy across all foreign langua *es, e.g., German, French). In addition to 
these modifications, minor wording changes occurred; the order of presentation of some of the 
knowledge and ability statements was changed, and some additional statements were included. 
The revised domain consisted of seven major content areas: (1) Listening Comprehension, (2) 
Reading Comprehension, (3) Structure of the Spanish Language, (4) Speaking, (5) Writing, (6) 
Culture (changed from Cultural Background), and (7) Foreign Language Pedagogy. Included 
within this domain were 166 individual knowledge and ability statements. In addition, a 
statement was added at the conclusion of each major content area section that assessed the 
overall importance of the content area. 

Pilot Test of the Job Analysis Survey 

Prior to the national administration, the job analysis survey was mailed to a small group 
of Spanish teachers and teacher educators. These pilot participants were asked to review the 
survey for clarity of wording and instruction, ease of use, and comprehensiveness of content 
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coverage. They were asked to make their comments on a questionnaire that accompanied the 
survey and to mail the questionnaire and survey back to ETS in a postage-paid envelope. No 
significant revisions were suggested by the pilot participants. 

Final Survey Forma t 

The finalized job analysis survey (see Appendix C for a copy of the survey) consisted of 
five sections. The first section included the seven content areas, the 166 specific knowledge and 
ability statements, and the seven overall importance statements. The second section provided 
the survey participants an opportunity to write in additional content areas and/or knowledge 
and ability statements that they believed should be considered part of the Spanish domain. The 
third section asked the survey participants to indicate, using a 5-point rating scale, how well each 
major content area covered the important knowledge and ability statements (an indicator of 
content coverage). The fourth section of the survey asked the participants to indicate the 
relative weight that each of the major content areas should receive on the examination. This 
was accomplished by their distributing 100 total points across the major areas. These point 
distributions were easily converted into percentages, representing the percent of items that the 
survey respondents believed should be devoted to each area. The last section was the 
background information section. The survey participants were asked to respond to several 
questions that described their demographic makeup (e.g., teaching experience, age, sex, 
race/ethnicity). This information was used to describe the survey respondents and to perform 
relevant subgroup analyses. These subgroup analyses will be described in the results section of 
this report. 

The knowledge and ability statements were judged using the following importance scale: 

How important is it for newly licensed (certified) Spanish teacher to know or be able to do 
each of the following in order to perform their jobs in a competent manner? 



This scale is consistent with the Standards emphasis on identifying a content domain that 
is important for competent job performance. This rating scale was reviewed and approved by 
the Advisory/Test Development Committee. 

Administration of the Job Analysis Survey 

The job analysis survey, accompanied by a letter of invitation to participate (see 
Appendix D for a copy of the letter), was mailed to 838 educational professionals across all 50 
states and the District of Columbia. The 838 educational professionals represented: teachers 
(n=*508), teacher educators (n = 252), and state administrators (n = 78). These distributions 
approximate the selection of 10 teachers, 5 teacher educators, and 1 state administrator for each 
state and the District of Columbia. These individuals were randomly selected from the 
memberships of the American Association of Teachers of Spanish and Portuguese, American 



(0) 
(1) 
(2) 
(3) 
(4) 



Of no importance 
Of little importance 
Moderately important 
Important 
Very important 



5 



16 



Council on the Teaching of Foreign Languages, and National Council of State Supervisors of 
Foreign Languages. These are the major professional organizations of Spanish teachers, teacher 
educators, and state administrators. Approximately 1 week after the surveys were mailed, a 
follow-up postcard was mailed to the participants reminding them to complete and return the 
surveys. 

The purpose of the survey administration was to identify a core of knowledge and ability 
statements that relatively large numbers of educational professionals judged to be relevant 
(verified to be important) to newly licensed (certified) Spanish teachers. The latter objective is 
accomplished by analyzing the mean importance ratings provided by the three groups of 
educational professionals and by the appropriate subgroups of respondents. Knowledge and 
ability statements that are judged to be important by all respondent groups and subgroups 
define the core. The core becomes the primary database for the development of test 
specifications. The derivation of test specifications from those knowledge and ability statements 
verified to be important by the surveyed educational professionals provides a substantial 
evidential basis for the content validity (relevance) of the Spanish subject assessments. 

Results 

Data Analyses of Survey Responses 

Two types of data analysis were conducted to support the development of content valid 
(content relevant) test specifications for the Spanish subject assessments: (1) means were 
computed of the importance ratings for each knowledge and ability statement by the three 
groups of educational professionals and by the appropriate subgroups of respondents; and (2) 
correlations of the profiles of these mean importance ratings were computed across the three 
groups of educational professionals and within the appropriate subgroups of respondents. 

Means . The mean analysis is used to determine the level (absolute value) of importance 
attributed to the 166 knowledge and ability statements by teachers, state administrators, and 
teacher educators and by appropriate subgroups of respondents (sex, race/ethnicity, geographic 
region, teaching experience). A respondent category was required to have at least 30 
respondents to be included in the mean analysis (e.g., > 30 state administrators, > 30 females). 
This, according to Walpole (1974), is a necessary condition to ensure that the mean value based 
upon the sample of respondents is an accurate estimate of the corresponding population mean 
value. Knowledge and ability statements that meet or exceed a mean importance value of 2.50 
(to be discussed in a later section) by all three groups of educational professionals (teachers, 
state administrators, teacher educators) and by all subgroups of respondents may be included in 
the development of the test specifications. In addition, mean ratings were computed for the 
responses to the content coverage section and the recommendation for test content section of 
the job analysis survey. These mean analyses were computed for the aggregate of the survey 
respondents to provide overall indicators. 

Correlations . The correlational analysis is used to determine the extent of agreement 
among the three groups of educational professionals and within the subgroups of respondents 
about the relative importance of the knowledge and ability statements. Relative importance 
refers to the similarity of the pattern of mean ratings generated by the different respondent 
groups. For example, the profile of 166 mean ratings for teachers is correlated with the profile 



of 166 mean ratings for state administrators. If these two profiles are similar (the shapes of the 
profiles are complementary), the value of the correlation coefficient will be close to 1.00. 

Criterion for Interpretation of Mean Importance Ratings 

Since the purpose of job analysis is to ensure that only the more important knowledge 
and ability statements are included in the development of test specifications, a criterion 
(cutpoint) for inclusion needs to be established. A reasonable criterion that has been used in a 
similar job analysis study (Rosenfeld & Tannenbaum, 1991) is a mean importance rating that 
represents the midpoint between moderately important and the next higher scale value. For the 
importance rating scale used in the present job analysis, the value of this criterion is 2.50 
(midpoint between moderately important and important). It is believed that this criterion is 
consistent with the intent of content validity, which is to include important knowledge and 
abilities in the assessment measure and to exclude unimportant knowledge and abilities from the 
assessment measure. Therefore, knowledge and ability statements that receive a mean 
importance rating of 2.50 or more may be considered eligible for inclusion in the development 
of test specifications; knowledge and ability statements that receive a mean rating of less than 
2.50 may not be considered for inclusion. (However, because survey participants were not 
involved in the development of the content domain, they may lack certain insights that the 
Advisory /Test Development Committee members have due to their high level of involvement in 
the definition of the domain. As a consequence, if the committee believes that a knowledge or 
ability statement rated below 2.50 should be included in the specifications and the committee 
can provide compelling written rationales, those knowledge and ability statements may be 
reinstated for inclusion in the test specifications.) 

Survey Respondents 

Response rate . Of the 838 total surveys that were mailed, nine were returned due to an 
invalid mailing address. Thus, 829 surveys were actually administered. Of these 829, 423 were 
returned. This represents an overall response rate of 51% (423/829). 

Demographic characteristics of the aggregate of the survey respondents . Seventy-five 
percent of the respondents were between the ages of 35 years and 54 years. Thirty percent were 
males and 70% were females. The majority of respondents (81%) were White. Most (68%) 
had 11 or more years of teaching experience in Spanish. The largest proportion of respondents 
(60%) were secondary school teachers; approximately 26% of the respondents were college 
faculty. Twenty-five percent of the respondents were from the Northeast region of the country; 
25% were from the Central region; 27% were from the Southern region; and 23% were from 
the Far West region. A complete breakdown of the demographic characteristics of the 
respondents is provided in Appendix E. 

Results of Data Analyses: Teachers. State Administrators, and Teacher Educators 

Mean importance . The mean importance rating for each of the 166 knowledge and 
ability statements is provided in Appendix F. The means are presented for (1) teachers 
(n = 237), (2) state administrators (n = 30), and (3) teacher educators (n= 104). Inspection of 
these distributions of mean ratings revealed that teachers rated 69% of the knowledge and 
ability statements (n = 115) 3.00 (important) or higher; state administrators also rated 115 
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statements (not necessarily the same ones) 3.00 or higher. Teacher educators rated 60% of the 
statements (n» 100) 3.00 or higher. A comparison of the means across the three groups 
revealed that state administrators rated 148 knowledge and ability statements (89%) higher than 
did teachers, and 151 (91%) higher than did teacher educators. Teachers rated 100 knowledge 
and ability statements (60%) higher than did teacher educators. 

The overall mean importance ratings for the seven major content areas were also 
computed for teachers, state administrators, and teacher educators. The means are presented in 
Table 1. All the means exceeded 3.00. This indicates that each major content area was judged 
to be important by each of the three groups of respondents. From a relative perspective, 
Speaking, Structure of the Spanish Language, and Listening Comprehension were among the 
highest rated areas; Culture and Writing were among the lowest rated areas. 



Table 1 

Overall Mean Importance Ratings for Each of the Seven Major Content Areas by Teachers, State 
Administrators, and Teacher Educators 



Means 


Major Content Areas 


Teachers 


State 
Administrators 


Teacher 
Educators 


Listening Comprehension 


3.46 


3.86 


3.57 


Reading Comprehension 


3.26 


3.60 


3.41 


Structure of the Spanish Language 


3.57 


3.70 


3.51 


Speaking 


3.63 


3.90 


3.60 


Writing 


3.27 


3.53 


3.22 


Culture 


3.07 


3.48 


3.15 


Foreign Language Pedagogy 


3.30 


3.87 


3.29 



As previously discussed, knowledge and ability statements that received a mean 
importance rating of less than 2.50 (midpoint between moderately important and important) 
may not be considered for inclusion in the development of test specifications, unless a 
compelling written rationale is provided by the committee for their reinstatement. Those 
knowledge and ability statements rated less than 2.50 by any one of the three respondent groups 
are presented in Table 2. Of the 166 individual knowledge and ability statements, 15 (9%) were 
rated below 2.50; however, none was rated below 2.00 (moderately important). 

Correlation of the profiles of mean importance rating s. The profiles of mean 
importance ratings for each of the three groups of educational professionals: (1) teachers, (2) 
state administrators, and (3) teacher educators were correlated. As noted above, the obtained 
correlation coefficients provide measures of the agreement among the respondent groups in 
terms of the relative importance of the knowledge and ability statements. The coefficients are 
presented in Table 3. All the values exceeded .85. This indicates that there is a high level of 
agreement among the three groups of educational professionals. 
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Table 2 

Knowledge and Ability Statements Rated Below 2SO by Teachers, State Administrators, and/or Teacher 
Educators 



Means 



Knowledffe and Ahilttv Statements 


Teacher* 


State 

A dm ini^tratnrc 


Teacher 

hps tore 


10. lucniuy assuinpuons on wnicn logical reasoning is oascu 


9 /11 
Z.41 






20. Understand utterances with background interference 






Z.4o 


zj. unuersianu speecn ai a rapid rate 


9 10 




ZOO 


Z*+. L/CHJOllolIalC aWaTCDCoS OI regional Variation 


9 4fl 




Z.«ri 


aH Trl#»ntift/ u/hsit tKp silltHrkr acciltnpc that n rpjl/tpr Irnnwc 


9 df) 


9 in 

Z.t/ 




«*z. ucicriuLnc me meaning oi figurative language 


9 dA 






/o. unuersianu luncuon oi uenvauves irom oiner languages 


1 It 

Z.i-3 


9 1*7 
Z.J/ 


Z.Uj 


/ /. unacrsiana uasic elements oi oral discourse 


9 VI 




o /1 1 
Z.41 


124. Demonstrate familiarity with contributions of groups 


2.35 




2.39 


127. Know about significant figures traditional 


2.43 






128. Know about significant figures contemporary 


2.24 




2.49 


129. Know significant styles of theater, music, art, etc. 


236 




2.34 


140. Give a rationale for teaching a particular content area 


2.36 




2.28 


141. Evaluate the impact of learning theories 






2.48 


172. Know effects of biculturalism, etc. 


2.48 




2.48 


Table 3 

Correlations of the Profiles of Mean Importance Ratings: Teachers, State Administrators, and Teacher 
Educators 


1 


2 




3 


1. Teachers 


.89 




.96 


2. State Administrators 






.87 


3. Teacher Educators 









ResultsjpiData Analyses: Subgroups of Respondents 



Mean importance . A significant contribution towards the accumulation of evidence in 
support of the job-relevance of the Spanish subject assessments is the verification of the 
importance of the knowledge and ability statements by a diversity of Spanish educational 
professionals. Therefore, mean importance ratings for each knowledge and ability statement 
were computed for the following subgroups of respondents: (1) race/ethnicity (Hispanic, n = 61; 



White, n=328); (2) sex (female, n*288; male, n-121); (3) geographic region (Northeast, n»100; 
Central, n»101; Southern-, n= 109; Far West, n»94), and (4) teaching experience 1 (s 5 years, 
n=33; > 5 years, n=202). The means are presented in Appendix G. An analysis of importance 
ratings by geographic region is consistent with recent legal emphasis on addressing regional job 
variability when conducting job analysis for content domain specification purposes (Kuehn et al., 
1990). The four geographic regions included for analysis are consistent with the categorization 
established by the National Association of State Directors of Teacher Education and 
Certification (NASDTEC). Sex and race/ethnicity subgroups were included because they 
represent protected "classes" under Title VII of the Civil Rights ACT of 1964. (Only Hispanic 
and White respondents were included because of the insufficient number of respondents, i.e., 
< 30, representative of other races/ethnicities.) The dichotomous breakdown of teaching 
experience at the 5-year point was chosen so that the judgments of less experienced teachers will 
be represented and so that the judgr ents of more experienced teachers will be represented. 

The results of the subgroup analysis by race/ethnicity, sex, and geographic region did not 
uncover any additional knowledge and ability statements (i.e., beyond the 15 statements 
previously identified by the mean analysis conducted on teachers, state administrators, and 
teacher educators) judged to be below the 2.50 cutpoint. However, the subgroup analysis 
conducted on teaching experience (< 5 years, > 5 years) did uncover six additional knowledge 
and ability statements judged to be below 2.50. These six statements were judged to be below 
2.50 only by the teachers with five or less years of teaching experience. This finding is 
consistent with the general tendency of the more experienced teachers to provide higher mean 
ratings than the less experienced teachers: more experienced teachers ( > 5 years) rated 125 of 
the 166 knowledge and ability statements (75%) higher than did less experienced teachers (s5 
years). The six statements and their means are presented in Table 4. 

Table 4 

Knowledge and Ability Statements Rated Below 230 by Teachers with Five or Less Years of Experience 



Knowledge and Ability Statements Mean 



10. 


Understand main idea of formal extended discourse 


2.45 


15. 


Determine if statements are presented as facts/opinions 


2.45 


102. 


Write comprehensive summaries 


2.39 


107. 


Restate and compare differing points of view 


2.39 


109. 


Analyze ideas and information in various ways 


2.31 


110. 


Produce and develop supporting material 


2.42 



Teaching experience includes only those respondents who had identified themselves as either a permanent substitute teacher or a 
regular teacher (n»23S). 



Correlations of the profiles of mean importance ratings . Correlation coefficients were 
computed for the profiles of mean importance ratings for the following subgroups of 
respondents: (1) geographic region (Northeast, Central, Southern, Far West), (2) teaching 
experience (< 5 years, > 5 years), (3) race/ethnicity (Hispanic, White), and (4) sex. The 
coefficients are presented in Table 5. All the values exceeded .90. This indicates that there is a 
very high level of agreement within the subgroups of respondents in terms of the relative 
importance of the knowledge and ability statements. 

Table 5 

Correlations of the Profiles of Mean Importance Ratings: Geographic Region, Teaching Experience, 
Race/Ethnicity, and Sex 

— _ __ — - — — 



GEOGRAPHIC REGION 

1. Northeast .94 .95 .97 

2. Central .98 .95 

3. Southern .% 

4. Far West 

TEACHING EXPERIENCE (years) 

1. £ 5 .94 

2. > 5 



RACE/ETHNICITY 

1. Hispanic .% 

2. White 

SEX 

1. Female .95 

2. Male 

Mean Ratings of Conte nt Coverage: Aggregate of Survey Respondents 

The survey participants were asked to indicate, using a 5-point rating scale, how well 
each of the seven major content area covered the important knowledge and ability statements. 
Responses to this provide an indication of the adequacy (comprehensiveness) of the domain of 
knowledge and abilities. The scale values ranged from a low of 1 (very poorly) to a high of 5 
(very well); the midpoint of the scale was value 3 (adequately). The means of these ratings for 
the aggregate of the survey respondents are presented in Table 6. All the mei n ratings exceed 
4.00. This indicates that all the major content areas were judged to have been weu covert.; 4 , and 
that the domain was comprehensive. 
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Table 6 

Mean Ra tings of Content Coverage 

Major Content Areas Mean 

Listening Comprehension 4.58 
Reading Comprehension 4.52 
Structure of the Spanish Language 4.48 
Speaking 4 - 56 
Writing 4 - 47 
Culture 4 -32 
Foreign Language Pedagogy 4 - 41 



Mean Percentage Weights for Test Content Emphasis: Aggregate of S urvey Respondents 

The survey participants were asked to indicate the weight that each of the seven major 
content areas should receive on the examination. This information may be used by test 
development committees to assist in their decisions about how much emphasis the content areas 
should receive on the test specifications. To obtain the weights, the participants were asked to 
distribute a total of 100 points across the major areas. The mean values were then converted 
into percentages. The mean percentage weights for the aggregate of the survey respondents are 
presented in Table 7. Speaking and Structure of the Spanish Language received the most 
emphasis (17% each). Culture and Foreign Language Pedagogy received the least emphasis 



(11% each). 
Table 7 

Mean Percentage Weights 

Major Content Areas Mean 

Listening Comprehension 15.69 
Reading Comprehension 14.06 
Structure of the Spanish Language 17.10 
Speaking 17-45 
Writing 12.51 
Culture 11-61 
Foreign Language Pedagogy H-68 



Summary and Conclusion 

A job analysis was conducted to define a knowledge and ability domain important for 
newly licensed (certified) Spanish teachers to perform their jobs in a competent manner. The 
results of the job analysis will be used to develop test specifications for the Spanish subject 
assessments of The Praxis Series: Professional Assessments for Beginning Teachers™. 
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An initial draft domain of important knowledge and ability statements was constructed 
by ETS Test Development staff with expertise in Spanish and ETS Research staff with expertise 
in job analysis. This draft domain was reviewed by an External Review Panel of subject-matter 
experts and then revised accordingly. The revised draft was then reviewed, modified, and 
approved by an external Advisory/Test Development Committee. The revised knowledge and 
ability domain was then subjected to verification/refutation through the use of a national survey 
of Spanish teachers, teacher educators, and state administrators. The survey participants were 
asked to rate the specific knowledge and ability statements of the domain using a 5-point 
importance scale. A cutpoint of 2.50 (midpoint between moderately important and important) 
was established to designate knowledge and ability statements as eligible (> 2.50) or ineligible 
( < 2.50) for inclusion in the development of test specifications. 

The results of the mean analysis conducted by teachers, state administrators, and teacher 
educators revealed 15 knowledge and ability statements that were rated less than 2.50. This 
represents only 9% of the content domain. Six additional knowledge and ability statements were 
rated below 2.50 by the subgroup of teachers with five or less years of experience. In total, only 
21 of the 166 statements (13%) did not meet the 250 criterion for inclusion. These 21 
knowledge and ability statements, previously presented with their associated mean values in 
Tables 2 and 4, have been combined for ease of identification in Table 8. 

The computation of correlation coefficients to assess agreement in terms of perceived 
relative importance of the knowledge and ability statements revealed a very high level of 
agreement. The coefficients for comparisons among teachers, state administrators, and teacher 
educators all exceeded .85; and the coefficients generated by the subgroup analyses all exceeded 
.90. These findings, coupled with the outcomes of the mean analyses indicate that there is 
substantial agreement in the importance ratings across the diversity of educational 
professionals. 

The 145 knowledge and ability statements that were verified to be important by the 
surveyed teachers, state administrators, teacher educators, and the subgroups should be us id 
as the foundation for the development of test specifications. Test specifications that are linked 
to the results of a job analysis provide support for the content validity of the derived assessment 
measures and may be considered as part of an initial step in ensuring the fairness (to subgroups 
of Spanish teacher candidates) of the derived assessment measures. It is reasonable to assume 
that, due to testing and psychometric constraints (e.g., time limits, ability to measure reliably 
some content), not all of the verified content may be included on the assessment measures. One 
source of information that may be used to guide the Test Development Committee in their 
decision of what verified content to include on the assessment measures is the mean importance 
rating. Although a rank ordering of the content by mean importance rating is not implied, it is 
recommended that initial consideration be given to content that is well above the cutpoint and 
represents the appropriate breadth of content coverage. 

Evidence was also provided in this study of the comprehensiveness of the content 
domain and the judged importance of the seven major content areas. These two pieces of 
information have implications for the adequacy of the content domain. If the domain was 
adequately defined then the knowledge and ability statements should be judged to have been 
well covered by each major content area and each major content area should be judged to be 
important. The results support the adequacy of the defined content domain. The knowledge 
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and ability statements were judged to have been well covered (a value of 4.0 on the content 
coverage scale); and the major content areas were judged to be important (a value of 3.0 on the 
importance scale). 



Table 8 

Twen ty-one Knowledge and Ability Statements Rated Below 2.50 

Knowled ge and Ability Statements 

10. Understand main idea of formal extended discourse 

15. Determine if statements are presented as facts/opinions 

16. Identify assumptions on which logical reasoning is based 

20. Understand utterances with background interference 

23. Understand speech at a rapid rate 

24. Demonstrate awareness of regional variation 

40. Identify what the author assumes that a reader knows 
42. Determine the meaning of figurative language 

76. Understand function of derivatives from other languages 

77. Understand basic elements of oral discourse 
102. Write comprehensive summaries 

107. Restate and compare differing points of view 

109. Analyze ideas and information in various ways 

110. Produce and develop supporting material 

124. Demonstrate familiarity with contributions of groups 

127. Know about significant figures traditional 

128. Know about significant figures contemporary 

129. Know significant styles of theater, music, art, etc. 

140. Give a rationale for teaching a particular content area 

141. Evaluate the impact of learning theories 
172. Know effects of biculturalism, etc. 
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Dr. Kenneth Tabor 
916 Woodlawn 
Ludington, MI 49431 



Ms. Ann Chance 
273 Princess Lane 
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Mr. Raul Rodriguez 
Language Department 
Xaverian High School 
7100 Shore Road 
Brooklyn, NY 11209 



Ms. Maria Ingham 
4014 Elizabeth Lane 
Fairfax, VA 22032 



Teacher Educators 



Dr. Alan Ga'iinkel 

Dept. of Foreign Languages and Literatures 

Stanley Coulter Hall 

Purdue University 

West Lafayette, IN 47907 



Dr. Karen Jogan 

Dept. of Modern Foreign Languages and Literatures 
Albright College 
P.O. Box 15234 
Reading, PA 19612-5234 



Dr. Richard Teschner 
Dept. of Languages and Linguistics 
University of Texas, El Paso 
El Paso, TX 79968 
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Dr. James Chatham 
Mississippi State University 
P.O. Box 6349 
Mississippi State, MS 39762 



Dr. Judith Liskin-Gasparro 
Dept. of Spanish and Portuguese 
Middlebury College 
Middlebury, VT 05753 
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A3 



29 



9 

ERIC 



Appendix B 

Advisory /Test Development Committee Members 



9 

ERIC 



B1 



Secondary School Teachers 
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Finneytown High School 
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Ms. Martha Mentch 
Albuquerque Academy 
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Ms. Dorian Pinner 
Washington High School 
Indianapolis, IN 46208 



Ms. Carol L. Sparks 
Foothill Middle School 
2775 Cedro Lane 
Walnut Creek, CA 94598 



Teacher Educators 



Dr. Orlando Edreira 

Kean College of New Jersey 
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West Virginia University 
Department of Foreign Language 
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INTRODUCTION 



Educational Testing Service (ETS) is developing a new generation of assessments for the 
purpose of licensing (certifying) teachers. The inventor; that follows is part of our development 
effort and is designed to gather information concerning the entry-level Spanish teacher's job . It 
was developed by high school teachers, college faculty, and state department of education 
officials, along with ETS staff. 

Those who constructed this inventory recognize that Spanish teachers are required to teach 
students with varying backgrounds and levels of ability. For these reasons, the collaborators 
believe that teachers should have a broad and deep understanding of Spanish in order to teach 
it. The inventory asks you to respond to a list of knowledge and ability statements and to rate 
each statement as to its importance for a newly-licensed (certified) Spanish teacher . Please do 
not relate each statement to your own job but rather to what you believe an entry-level Spanish 
teacher should know and be able to do. 

The information you provide will guide the development of the new NTE Spanish 
examination. It is expected that the new examination will differ from the current examination in 
both content and design. In addition to the development of a new examination, this study will 
also contribute to our understanding of Spanish teaching as a profession. We expect the results 
of the study to be widely disseminated and to have ramifications for teacher preparation. 

The inventory has been mailed to a sample of 800 professionals. Its value is directly related 
to the number of individuals who return their completed inventories. Because you represent a 
large number of professionals, your responses are extremely important. Please take the time to 
complete and return the inventory. 
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INVENTORY OF KNOWLEDGE AND ABILITIES 
FOR SPANISH TEACHERS 



The purpose of this inventory is to determine what you believe newly-licensed (certified) 
Spanish teachers should know and be able to do in order to perform their job in a competent 
manner. On the following pages you will find seven major categories (content areas) and 
beneath each, a list of knowledge and ability statements that define the particular content area. 
The order of presentation of the seven content areas in the inventory is not meaningful. 

The seven content areas are: 

(A) Listening Comprehension 

(B) Reading Comprehension 

(C) Structure of the Spanish Language 

(D) Speaking 

(E) Writing 

(F) Culture 

(G) Foreign Language Pedagogy 

For each statement within each of these content areas, you will be asked to make the following 
judgment: 



How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers to 
know or be able to do the following in order to perform their job in a competent 
manner? 

Circle your response using the 5-point scale adjacent to each statement. 



(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



To familiarize yourself with the content areas and sta v '.=ments you may wish to briefly glance 
through the inventory before making your judgments. 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



A. Listening Comprehension; The ability to understand Spanish when spoken by IMPORTANCE 
an educated native speaker at a normal rate of speech. 

1. Recognize the meaning carried by stress and intonation 

(e.g., esta vs. esta" / tVas manana? vs. Vas manana.) 0 1234 

2. Extract meaning of an utterance based on key words 0 1 2 3 4 

3. Understand vocabulary and sentences from their context 0 1234 

4. Understand high frequency vocabulary spoken in context 0 1234 

5. Understand high frequency idiomatic expressions and figures of 

speech 0 j 2 3 4 

6. Recognize what question has been asked or what instruction has 

been given 0 j 2 3 4 

7. Understand simple, complex, and compound sentences in past, 

present, and future q j 2 3 4 

8. Understand sentences with more complex constructions 

(e.g., contrary-to-fact clauses) q \ 2 3 4 

9. Understand formal exchanges between or among other people 
(e.g., listening to a television or radio interview or panel 

discussion) q j 2 3 4 

10. Understand the main ideas of formal extended discourse such as 

speeches and lectures on non-technical topics 0 12 3 4 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



Listening Co m prehension (continued) IMPORTANCE 

11. Understand the main ideas of non-technical broadcasts (e.g., radio, 

TV -- advertisements, announcements, news broadcasts) 0 1234 

12. Identify the main idea of an informal conversational exchange when 

it is stated directly 0 1 2 3 4 

13. Identify the main idea of a conversational exchange when it can be 

inferred on the basis of information in the conversation 0 1234 

14. Recognize internal inconsistencies in a spoken message (e.g., when a 
speaker gives contradictory information or makes statements that do 

not coincide with what the speaker said earlier) 0 1 2 3 4 

15. Determine whether statements in a spoken message are presented 

as facts or opinions 0 1 2 3 4 

16. Identify assumptions on which the logical reasoning in a spoken 

message is based 0 1 2 3 4 

17. Identify thj affective or emotional content of a spoken message 

(e.g., affection, sadness, humor, irony, sarcasm) 0 1 2 3 4 

18. Recognize different uses of speech in different social situations ... 01234 

19. Understand routine telephone exchanges 0 1 2 3 4 

20. Understand utterances with some background interference 0 12 3 4 

0 12 3 4 

21. Understand conversations on a variety of everyday topics 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



A. Listening Comprehension (continued) IMPORTANCE 

22. Understand speech at a normal conversational rate 0 1 2 3 4 

23. Understand speech at a rapid rate 0 1 2 3 4 

24. Demonstrate an awareness of regional variations of pronunciation 
and vocabulary (e.g., Mexican, Argentinean, Caribbean or 

Iberian Spanish) 0 1 2 3 4 

25. Overall evaluation of the importance of Listening 

Comprehension 0 1 2 3 4 

B. Reading Comprehension : The ability to understand written Spanish. 
Jnderstand the content and organization of a reading selection 

26. Determine the main theme 0 1 2 3 4 

27. Identify important ideas 0 1 2 3 4 

28. Identify the supporting details 0 1 2 3 4 

29. Identify paraphrases or summaries of ideas 0 1 2 3 4 

30. Identify the relationships among ideas directly stated 

(e.g., relationships of cause and effect and sequence) 0 1 2 3 4 

31. Locate the place in a reading selection where a specific kind of 

information can be found 0 1 2 3 4 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 



Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



Understand the implied content of a reading selection IMPORTANCE 

32. Distinguish what is implied from what is directly stated 0 1234 

33. Make inferences from the directly stated content 0 1 2 3 4 

34. Recognize the attitude(s) expressed (e.g., subjective vs. 

objective tone) 0 1 2 3 4 

35. Recognize the style or manner of expression (e.g., formal vs. 

informal) 0 1 2 3 4 

36. Distinguish fact from opinion 0 1 2 3 4 

37. Determine the author's purpose for writing the selection 0 1234 

38. Predict the logical sequence of ideas or the outcome of events ... 01234 
Understand the wav in which language k used in a reading selection 

39. Recognize different interpretations of a word, sentence, paragraph, 

or the entire selection 0 1 2 3 4 

40. Identify what the author assumes that a reader knows when a 

specific word or phrase is used 0 1 2 3 4 

41. Recognize how the meaning of a word, sentence, or paragraph is 

affected by the context in which it appears 0 1 2 3 4 

42. Determine the meanings of figurative language (e.g., metaphors, 0 12 3 4 
similes) 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 

Understand the wav in which language is used in a reading selection (cont.) IMPORTANCE 

43. Understand the function of key transition indicators (e.g., "sin 
embargo", "no obstante", "con todo", "por eso","por !o tanto", "por 

consiguiente...") 0 1 2 3 4 

44. Understand a variety of materials commonly read by the general 

public (e.g., newspapers, magazines) 0 1 2 3 4 

45. Overall evaluation of the importance of Reading 

Comprehension 0 1 2 3 4 

C. Structure of the Span ish Language; Demonstrate the correct formation and 
use of grammatical structure and appropriate style in written and 
spoken Spanish. 

Demonstrat e the correct formation and users) of . . . 

46. regular and irregular verbs (including orthographic- and 

stem-changing) in all indicative tenses 0 1 2 3 4 

47. regular and irregular verbs in the conditional 0 1 2 3 4 

48. regular and irregular verbs (including orthographic- and 

stem-changing) in the subjunctive mood 0 i 2 3 4 

49. the formal and informal commands (both regular and irregular) . . 0 12 3 4 

50. reflexive verbs 0 1 2 3 4 

51. ser vs. estar 0 1 2 3 4 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



Demonstrate the correct formation and use(s) of . . . (cont.) IMPORTANCE 

52. saber vs. conocer 0 1 2 3 4 

53. the preterite vs. the imperfect 0 1 2 3 4 

54. expressions of futurity (e.g., ir a + infinitive, 

pensar + infinitive) 0 1 2 3 4 

55. future and conditional of probability 0 1 2 3 4 

56. hace. . .que expressions in the present and past 0 1 2 3 4 

57. regular and irregular present participles 0 1 2 3 4 

58. regular and irregular past participles 0 1 2 3 4 

59. the passive voice and its reflexive or active voice substitutes 

(e.g., se vende) 0 1 2 3 4 

60. personal pronouns (subjective, objective; direct, indirect; 

prepositional and reflexive) 0 1 2 3 4 

61. relative pronouns (e.g., que, el que, el cual, lo que, quien, cuyo) . . 0 12 3 4 

62. demonstrative and possessive adjectives and pronouns 0 1 2 3 4 

63. conjunctions 0 1 2 3 4 

64. interrogatives and exclamations 0 1 2 3 4 

65. comparatives and superlatives 0 1 2 3 4 



C10 

42 

a 

ERIC 



How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



Demonstrate the correct formation and use(s) of . . . (cont.) 

66. plural forms of nouns and adjectives 

67. agreement of adjectives, nouns, verbs, and pronouns 

68. por vs. para 

69. definite and indefinite articles 

70. idiomatic expressions (e.g., tener que, hay que, dar con) 

71. expressions for describing weather (e.g., hace frfo) 

Language Analysis 

72. Understand the basic meaningful elements of words (morphemes) 
(e.g., augmentative, diminutive, and pejorative suffixes, 
tense/person verb endings, gender/number markers in nouns 
and adjectives, adverb formation) 

73. Understand word order to form phrases, clauses, or sentences 
(e.g., in affirmative and negative sentences, 
statements/questions/commands, placement of adjectives) 

74. Understand the formation of compound words and contractions 
(e.g., paraguas, al) 

75. Understand the function of cognates and false cognates in the 
development of vocabulary 



IMPORTANCE 

0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 



0 12 3 4 



0 12 3 4 



0 12 3 4 



0 12 3 4 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



Language Analysis (continued) IMPORTANCE 

76. Understand the function of derivatives from other languages in 

contemporary Spanish (e.g., algebra - Arabic, ch6fer - French) ... 01234 

77. Understand the properties of the basic elements of oral discourse 

(e.g., phonemes, points/manner of articulation, voicing) 0 12 3 4 

78. Demonstrate knowledge of stress, rhythm, and intonation patterns of 

standard Spanish and how these compare to English 0 1 2 3 4 

79. Identify common pronunciation problems for non-native speakers of 

Spanish (e.g., trill V vs, tap V) 0 1 2 3 4 

80. Overall evaluation of the importance of the Structure of the 

Spanish Language 0 1 2 3 4 

D. Speaking ; The ability to speak Spanish. 

81. Speak well enough to be intelligible to an educated native speaker of 

Spanish 0 1 2 3 4 

82. Converse in a clearly participatory fashion on a variety of concrete 

everyday topics 0 1 2 3 4 

83. Speak with appropriate stress, intonation and rhythm 0 1234 

84. Demonstrate a working vocabulary relevant to a variety of concrete 

and everyday topics 0 1 2 3 4 



9 

ERIC 



C12 



44 



How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



D. Speaking (continued) IMPORTANCE 

85. Speak with sufficient grammatical accuracy so as to be easily 
understood by Spanish speakers unaccustomed to speaking with 

foreigners 0 1 2 3 4 

86. Initiate, sustain, and bring to closure a wide variety of 
communicative tasks (e.g., explain, discuss, ask and respond to 

questions, request, praise, deny, interact socially) 0 1 2 3 4 

87. Present opinions on familiar topics 0 1 2 3 4 

88. Restate and compare differing points of view 0 1 2 3 4 

89. Defend opinions on familiar topics 0 1 2 3 4 

90. Narrate and describe situations with paragraph-length discourse 

using appropriate structures to convey past, present, and future ... 01234 

91. Adapt vocabulary and styk jf language to the situation and the 

audience (e.g., use suitably formal or informal language) 0 1234 

92. Supply a synonym for or explanation of a word/phrase that has not 

been understood 0 1 2 3 4 

93. Request clarification of a spoken message when it has not been 

understood 0 1 2 3 4 

94. Summarize orally a speaker's main ideas 0 1 2 3 4 

95. Read aloud with the intonation, inflection, and fluency necessary to 

communicate written material effectively 0 1 2 3 4 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 

D. Sesajuns (continued) IMPORTANCE 

96. Provide comprehensible directions/instructions to students 0 12 3 4 

97. Overall evaluation of the importance of Speaking 0 1 2 3 4 

E. Writing; The ability to write Spanish. 
Composing Skills 

98. Write well enough to be intelligible to educated native speakers of 

Spanish 0 1 2 3 4 

99. Use appropriate transitional expressions 0 1 2 3 4 

100. Write simple social correspondence 0 1 2 3 4 

101. Take notes 0 1 2 3 4 

102. Write comprehensive summaries 0 1 2 3 4 

103. Describe an event or situation of a factual nature 0 1 2 3 4 

104. Delineate the steps in a process or procedure (e.g., explain how to 

do something) 0 1 2 3 4 

105. Provide comprehensible directions/instructions to students 0 12 3 4 

106. Present an opinion on familiar topics 0 1 2 3 4 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



Composing Skills (continued) 

107. Restate and compare differing points of view 

108. Defend an opinion on familiar topics 

109. Analyze ideas and information in various ways (e.g., explain the 
cause and effect, draw conclusions) 

110. Produce and develop supporting material that explains or illustrates 
key ideas 

111. Organize ideas effectively and logically 

1 12. Demonstrate a working vocabulary relevant to a variety of concrete 
and everyday topics 

1 13. Use language appropriate to the task and/or the audience addressed 

1 14. Produce writing that is free of major errors in grammar, 
punctuation, syllabication, and spelling 

Editing and Revising Skills 

1 15. Recognize basic grammatical errors in written Spanish 

(e.g., in morphology, syntax) 

1 16. Understand why native English speakers are prone to make certain 
basic grammatical errors in written Spanish 



IMPORTANCE 

0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 

0 12 3 4 



0 12 3 4 



0 12 3 4 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



Editing and Revising Skills (continued) IMPORTANCE 

117. Recognize language that is inappropriate to the task and/or the 

audience addressed 0 1 2 3 4 

118. Revise writing that is inappropriate to the task and/or the audience 

addressed 0 1 2 3 4 

119. Revise writing to clarify meaning 0 1 2 3 4 

120. Overall evaluation of the importance of Writing 0 1 2 3 4 

F. Culture ; Knowledge of the society, history, and geography of Spanish-speaking 
regions and countries. 

121. Locate the Spanish-speaking regions and countries 0 1234 

122. Identify the major geographical features of Spanish-speaking regions 

and countries 0 1 2 3 4 

123. Demonstrate familiarity with the major works and authors of the 

literature of Spanish-speaking countries 0 1 2 3 4 

124. Demonstrate familiarity with the contributions of other linguistic 
and cultural groups to the culture of Spanish-speaking regions 
and countries (e.g., Basques, American Indian groups, Arabs, 

U.S.A.) 0 1 2 3 4 

125. Know about the major groups indigenous to the Spanish-speaking 

Americas 0 1 2 3 4 

126. Recognize the contributions of Spanish to other cultures (e.g., Meso- 

American and U.S. cultures) 0 1 2 3 4 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



F. Culture (continued) IMPORTANCE 

127. Know about significant figures and their works in traditional theater, 
music, art, and dance of Spanish-speaking countries (e.g., Lope 

de Vega, Goya, Juan Ruiz de Alarc6n) 0 1 2 3 4 

128. Know about significant figures and their works in contemporary 
theater, film, music, art, and dance of Spanish-speaking countries 

(e.g., Bunuel, Alicia Alonso, Ima Sumac) 0 1 2 3 4 

129. Know about significant styles of theater, music, art, architecture, and 

dance of Spanish-speaking countries 0 1 2 3 4 

130. Identify major (landmark) contributions of Spanish-speaking people 
to the history, economy, political life, and culture of the United 

States 0 1 2 3 4 

131. Know about the location and ethnic composition of the major 

populations of Spanish-speaking people in the United States 0 12 3 4 

132. Demonstrate an awareness of contemporary world politics and 
economics affecting Spanish-speaking countries (e.g., the 
O.E.A., political movements, the European Community, 

environmental concerns) 0 1 2 3 4 

133. Know about contemporary lifestyles (e.g., food, mealtimes, customs, 
holidays, sports, family relationships, religion, regional variations, 

foreign influences, monetary units) 0 1 2 3 4 

134. Recognize the socio-linguistic elements associated with the Spanish 
language (e.g., body language, use of tu vs. usted, social interaction 

patterns) 0 1 2 3 4 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 




to know or be able to do each of the following in order to perform their job 




in a competent manner? 




Circle your response using the 5-point scale adjacent to each statement. 




(0) OF NO IMPORTANCE 




(1) OF LITTLE IMPORTANCE 




(2) MODERATELY IMPORTANT 




(3) IMPORTANT 




(4) VERY IMPORTANT 





F. Cjilture (continued) IMPORTANCE 

135. Recognize the most important events and principal historic figures 

of Spanish-speaking countries 0 1 2 3 4 

136. Overall evaluation of the importance of Culture 0 1 2 3 4 

G. Foreign Language Pedagog y: This includes knowledge of students, curriculum, 
teaching strategies, and resources. 

137. Give a rationale for learning Spanish 0 1 2 3 4 

138. Explain the role of Spanish in the total curriculum 0 1 2 3 4 

139. Give a rationale for teaching a particular topic in Spanish 0 1234 

140. Give a rationale for teaching a particular content area 

(e.g., mathematics, science, language arts) in Spanish 0 1 2 3 4 

141. Evaluate the impact of learning theories on teaching Spanish 

(e.g., second language acquisition, developmental learning) 0 12 3 4 

142. Demonstrate an understanding of current methodologies 

(e.g., delayed production, proficiency-based approaches) 0 1234 

143. Evaluate the organization of content in a particular Spanish 

curriculum 0 1 2 3 4 

144. Demonstrate an awareness of the importance of articulation 
(coordination among elementary, middle, secondary school and 

college/university Spanish curricula) 0 1 2 3 4 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



G. Foreign Language Pedagogy (continued) IMPORTANCE 

145. Organize content in a Spanish curriculum for a specific group of 
students and justify it 0 1 2 3 4 

146. Identify the knowledge, experience, and skills that students bring to 
various content areas in Spanish 0 1 2 3 4 

147. Evaluate and make appropriate use of curricular materials and 
resources for Spanish (e.g., textbooks, authentic materials, 

computer software, laboratory materials, learning kits) 0 1234 

148. Evaluate and make appropriate use of mass media (e.g., film, 
television, video) to teach Spanish 0 1 2 3 4 

149. Evaluate and make appropriate use of instructional 
technologies (e.g., computer, videodisc, interactive 

television) to teach Spanish 0 1 2 3 4 

150. Evaluate teaching strategies and activities as related to specific 
objectives (e.g., laboratory work, supervised practice, group work, 

lecture) 0 1 2 3 4 

151. Select and use a variety of appropriate teaching strategies 0 12 3 4 

152. Use appropriate forms of representation (e.g., analogies, examples, 
visuals, concrete objects, realia) to make Spanish understandable 

and interesting 0 1 2 3 4 

153. Show relationships among topics in Spanish (e.g., similarities 
between endings in the imperfect and the conditional, Arab 

influence on Spanish history, language and art) 0 1 2 3 4 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



G. Foreign Language Pedagogy (continued) IMPORTANCE 

154. Integrate Spanish and other curricular areas 0 1 2 3 4 

155. Use students' out-of-class experiences and knowledge to teach 

s P anis h 0 1 2 3 4 

156. Construct appropriate instruments to evaluate students' 

performance in Spanish 0 1 2 3 4 

157. Use appropriate evaluation strategies (e.g., portfolios, 
observations, interviews, oral discussions, essays, 
written tests, diaries, daily logs) to assess students' 

performance in Spanish 0 1 2 3 4 

158. Develop a variety of questioning strategies that ask students to 

display their understanding of particular topics 0 1 2 3 4 

159. Engage students in developing their own questions about topics in 

s P anish 0 1 2 3 4 

160. Distinguish errors which impede comprehension from less serious 

°nes 0 1 2 3 4 

161. Assist students in developing strategies to overcome first-language 
interference and other error inducing linguistic patterns 

0 12 3 4 

162. Recognize patterns of error in students' work and assist students in 

developing strategies to recognize and correct them 0 1 2 3 4 

163. Manage student interaction in a Spanish classroom without 

discouraging divergent thought 0 1 2 3 4 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish (eachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 



Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



G. Foreign Language Pedagogy (continued) IMPORTANCE 

164. Demonstrate an awareness of strategies for motivating and 

encouraging students to succeed in Spanish 0 1 2 3 4 

165. Understand complex factors (e.g., societal, gender, racial, 

socio-economic, language) that affect the teaching of Spanish .... 01234 

166. Understand complex factors (e.g., societal, gender, racial, 
socio-economic, language) that affect students' learning 

of Spanish 0 1 2 3 4 

167. Demonstrate an awareness of professional and scholarly 
literature (e.g., journals, reference works, magazines) 

appropriate for Spanish teachers and students 0 1 2 3 4 

168. Demonstrate an awareness of professional and scholarly 
organizations for Spanish and other foreign language teachers and 

students 0 1 2 3 4 

169. Understand current language proficiency guidelines (e.g., ACTFL, 

FSI) '• • • 0 1 2 3 4 

170. Show students ways to study Spanish effectively 0 1 2 3 4 

171. Identify English language interference and other linguistic 
phenomena characteristic of native Spanish speakers studying 
Spanish and assist students to develop strategies to overcome 

them 0 12 3 4 
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How IMPORTANT is it for NEWLY-LICENSED (CERTIFIED) Spanish teachers 
to know or be able to do each of the following in order to perform their job 
in a competent manner? 

Circle your response using the 5-point scale adjacent to each statement. 

(0) OF NO IMPORTANCE 

(1) OF LITTLE IMPORTANCE 

(2) MODERATELY IMPORTANT 

(3) IMPORTANT 

(4) VERY IMPORTANT 



G. Foreign Language Pedagogy (continued) 



IMPORTANCE 



172. Know linguistic, social, and cultural phenomena associated 

with biculturalism and bilingualism in the U.S. and how these 
phenomena affect the teaching of Spanish to native Spanish 
speakers 



0 12 3 4 



173. Overall evaluation of the importance of Foreign Language 
Pedagogy 



0 12 3 4 
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ADDITIONAL COMMENTS 



Please use this space to list any important CONTENT AREAS that you believe were NOT 
included in this inventory. 



Please use this space to list any important STATEMENTS that you believe were NOT included 
in this inventory. For each statement please indicate the category to which it belongs. 
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CONTENT COVERAGE 



Hew well were the important knowledge and abilities covered in each of the content areas in the 
inventory? (Circle your response using the 5-point scale). 

A. Listening Comprehension 

1 2 3 4 5 

Very Poorly Poorly Adequately Well Very Well 



B. Reading Comprehension 

1 2 3 4 5 

Very Poorly Poorly Adequately Well Very Well 



C. Structure of the Spanish Language 

1 2 3 4 5 

Very Poorly Poorly Adequately Well Very Well 



D. Speaking 

1 2 3 4 5 

Very Poorly Poorly Adequately Well Very Well 



E. Writing 

1 2 3 4 5 

Very Poorly Poorly Adequately Well Very Well 



F. Culture 

1 2 3 4 5 

Very Poorly Poorly Adequately Well Very Well 



G. Foreign Language Pedagogy 

1 2 3 4 5 

Very Poorly Poorly Adequately Well Very Well 
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RECOMMENDATIONS FOR TEST CONTENT 



Listed below are the seven content areas that may be covered on the new NTE Spanish 
examination. If the examination were to contain 100 questions, how many questions do you 
believe should be included from each content area? 

Please indicate your response using whole numbers (no fractions). If you believe a content area 
should not be included in the examination, put a 0 in the space provided. Please make sure that 
your responses sum to 100. 

CONTENT AREAS NUMBER OF EXAM QUESTIONS 

(OUT OF 100) 

A. Listening Comprehension 

B. Reading Comprehension 

C. Structure of the Spanish Language 

D. Speaking 

E. Writing 

F. Culture 

G. Foreign Language Pedagogy 



TOTAL 100 
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BACKGROUND INFORMATION 



The information which you provide in this section is completely confidential and will be used for 
research purposes only. Your responses will be grouped statistically with those of other 
individuals who are participating in this survey. A vital part of the statistical analysis consists of 
grouping people with varying experience and varying backgrounds. To do this, we need your 
answers to the following questions. 

(Circle your response). 



174. In which state do you work? 



1. Alabama 

2. Alaska 

3. Arizona 

4. Arkansas 

5. California 

6. Colorado 

7. Connecticut 

8. Delaware 

9. District of 
Columbia 

10. Florida 

11. Georgia 

12. Hawaii 
13. Idaho 

14. Illinois 

15. Indiana 

16. Iowa 

17. Kansas 



18. Kentucky 

19. Louisiana 

20. Maine 

21. Maryland 

22. Massachusetts 

23. Michigan 

24. Minnesota 

25. Mississippi 

26. Missouri 

27. Montana 

28. Nebraska 

29. Nevada 

30. New Hampshire 

31. New Jersey 

32. New Mexico 

33. New York 

34. North Carolina 

35. North Dakota 



36. Ohio 

37. Oklahoma 

38. Oregon 

39. Pennsylvania 

40. Rhode Island 

41. South Carolina 

42. South Dakota 

43. Tennessee 

44. Texas 

45. Utah 

46. Vermont 

47. Virginia 

48. Washington 

49. West Virginia 

50. Wisconsin 

51. Wyoming 



175. What is your age? 

1. Under 25 

2. 25-34 

3. 35-44 

4. 45-54 

5. 55-64 

6. 65 and over 



176. What is your sex? 

1. Female 

2. Male 
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BACKGROUND INFORMATION (continued) 
177. Is your ethnic heritage Hispanic? 

1. Yes 

2. No 



178. Which of the following best describes you? 

1. American Indian or Alaska Native 

2. Black or African American 

3. Asian or Pacific Islander 

4. White 

5. Other 



179. Which of the following best describes your highest educational 
attainment? 

1. Less than a Bachelors degree 

2. Bachelors degree 

3. Bachelors degree + additional credits 

4. Masters degree 

5. Masters degree + additional credits 

6. Doctorate 



180. Which of the following best describes your current employment status? 

1. Temporary substitute (assigned on a daily basis) 

2. Permanent substitute (assigned on a longer term basis) 

3. Regular teacher (not a substitute) 

4. Principal or Assistant Principal 

5. School Administrator 

6. Curricular Supervisor 

7. State Administrator 

8. College Faculty 

9. Other (please specify) 



181. Are you a native Spanish speaker? 

1. Yes 

2. No 
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BACKGROUND INFORMATION (continued) 



182. How many years have you taught Spanish? 

1. Less than a year 

2. 1 - 2 years 

3. 3 - 5 years 

4. 6-10 years 

5. 11-15 years 

6. 16-20 years 

7. 21 or more years 

8. Never taught Spanish 



183. What is the predominant school level at which you teach? 
(Circle only one category) 

1. Elementary 

2. Middle 

3. Secondary 

4. College or University 

5. Do not currently teach 

6. Other (please specify) 



184. Circle the organizations to which you currently belong. 

1. American Association of Teachers of Spanish and Portuguese 

2. American Council on the Teaching of Foreign Languages 

3. National Council of State Supervisors of Foreign Languages 



THANK YOU FOR COMPLETING THIS INVENTORY. 
PLEASE RETURN IT WITHIN 10 DAYS USING THE ENCLOSED ENVELOPE. 
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Survey Cover Letter 
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KDL'CATIONAL TESTING SERVICE j' £ PRINCETON, N.J. 08541 



^'.VISION OF -UTUED 
■[LASUREMENT RESEARCH 



May 1, 1990 
Dear Colleague: 

I am writing to ask your cooperation in a project that should be of importance to 
teachers, college faculty, administrators, and other professionals in the field of Spanish. 
Educational Testing Service (ETS) is in the process of developing a new generation of 
assessments for the purpose of licensing (certifying) teachers. One type of assessment will be 
created to measure the prospective teacher's subject-matter or specialty-area knowledge and will 
likely take place upon completion of the undergraduate teacher education program. One such 
assessment is a new version of the NTE Spanish examination. I am asking for your help as we 
develop this examination. 

As part of the developmental process of the new NTE Spanish examination, ETS has 
worked closely with teachers, college faculty, and school administrators to identify potentially 
important knowledge areas and abilities. The enclosed inventory has been developed as a way 
to obtain your judgments on the importance of these knowledge and abilities for the newly- 
licensed (certified) Spanish teacher. The data obtained from these inventories will be used to 
guide the design and content of the new NTE Spanish examination. 

This inventory is being mailed to approximately 800 practicing professionals. Its value is 
directly related to the number of individuals who return their .inventories. Because you 
represent a large number of professionals, your judgments are ;'ery important. Your responses 
will be kept in strict confidence. The inventory asks for some background information about 
you; this is solely for purposes of describing the group of respondents. 

The results of this study are expected to be widely disseminated and should be very 
useful to the profession. 

A postage-paid envelope is enclosed for the return of your completed questionnaire. 
Thank you for your participation in this very important project. 

Sincerely, 



Richard Tannenbaum, Ph.D. 
Associate Research Scientist 
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Respondent Demographics 
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Number 


jrerccni 


GEOGRAPHIC REGION 






Northeast 


100 


24.7 


Central 


101 


25.0 


Southern 


109 


27.0 


Far West 


94 


Zj.Z 


Total 


404 




Missing Responses 


19 




AGE (years) 






Under 25 


2 


0J5 


25-34 


34 


83 


35-44 


172 


42.0 


45-54 


135 


32.9 


55-64 


58 


14.1 


Over 64 


9 


2.2 


Total 


410 




Missing Responses 


13 




SEX 






Female 


288 


70.4 


Male 


121 


29.6 


Total 


409 




Missing Responses 


14 




SCHOOL LEVEL 






Elementary 


2 


0.5 


Middle 


21 


5.2 


Secondary 


244 


60.2 


College or University 


107 


26.4 
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Number 


rercent 


SCHOOL LEVEL (cont.) 






Do not currently teach 


31 


7.7 


Total 


405 




Missing Responses 


18 




RACE/ETHNICITY 






Native American 


0 


0.0 


Asian American 


5 


1.2 


Black 


10 


2.5 


Hispanic 


61 


15.0 


White 


328 


80.6 


Other 


3 


0.7 


Total 


407 




Missing Responses 


16 




HIGHEST EDUCATIONAL ATTAINMENT 






Less than Bachelor's 


0 


0.0 


Bachelor's 


3 


0.7 


Bachelor's + Credits 


66 


16.1 


Master's 


44 


10.7 


Master's + Credits 


193 


47.1 


Doctorate 


104 


25.4 


Total 


410 




Missing Responses 


13 




CURRENT EMPLOYMENT STATUS 






Temporary Substitute 


2 


U.5 


Permanent Substitute 


1 


0.3 


Regular Teacher (not a substitute) 


236 


59.4 


Principal/Assistant Principal 


1 


0.3 


School Administrator 


0 


0.0 
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Number 


Percent 


niDDEMT CXjfDT nVMCMT CTATI TC (pnnt \ 
V.UKKC.IN 1 cmrLUIMcIN 1 o 1 A i Uo ^COul.j 






Curriculum Supervisor 


j 


0.8 


State Administrator 


30 


7.6 


College Faculty 


104 


26.2 


Miner 




5.0 


l Olai 






IVllSdlXlg KCSpuDSCS 


so 










in ever laugni 


14. 


3.4 


Less man 1 


I 


0.5 


1 . "5 
1 - Z 


1Z 


2.9 


1 < 
J - J 


JO 


8.8 


O 1U 


OS 


16.6 


11 - 15 


91 


22.2 


16-20 


73 


17.8 


zi or more 




27.8 


l Ula-l 








11 




NATIVE SPANISH SPEAKER 






Yes 


61 


14.9 


INO 


34a 


85.1 


Total 


409 




Missing Responses 


14 




MEMBERSHIP IN ORGANIZATIONS 






American Association of Teachers of Spanish and 
Portuguese (AATSP) 


167 


42.0 


American Council on the Teaching of Foreign 
Languages (ACTFL) 


49 


12.3 
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Number 


Percent 


MEMBERSHIP IN ORGANIZATIONS (cont.) 






National Council of State Supervisors of Foreign 




0.8 


Languages (NCSSFL) 


3 


AATSP & ACTFL 


156 


39.2 


AATSP & NCSSFL 


1 


0.3 


ACTFL & NCSSFL 


15 


3.8 


AATSP, ACTFL & NCSSFL 


7 


1.8 


Total 


398 




Missing Responses 


25 





E5 



67 



Appendix F 

Mean Importance Ratings: 
Teachers. State Administrators, and T eacher Educators 



9 

ERIC 



F1 



68 



in in 
o o 



>n 
>n 



8 



VO 
m 



co 
m 

o 



oo oo 
o o 



o 



« vo .-i >n 
vO 00 ov o 
O o o 



CO 

On 



r- 



o d 



ON 

o o o 



is 



o 
r-, 
co 



5 



00 

CO 



o 
in 
co 



co 



8 «q 

co co 



Ov 
CN 



co o\ 
o »-< 
co co 



cn 
co 



cn 
o 



co cn 



3 R 

CN CN 



m 
oo 

CN 



CO «-< 

oo <N 
cn co 



CN 



co 



5B 58 

o o 



CO 



3* 



>n 
to 



in 

CO 



58 S 

o o 



0\ 

c5 



0\ 
VO 

o 



«-l CO 
CO vO 

o o 



3 

o 



CO 

r- 
o 



>n 

Ov 

o 



o 



oo 
o 



in 



S 5 

o o 



in 

CO 



8$8££&&RrN:$ 

cocococococococococo 



cococococofNcocococococo 



CO CN "* 

vo vo in 
odd 



o 



d 



o 
» 
o 



d 



m 
oo 



d 



t-i cn 
00 vo 

d d 



f-M OV 

t— 00 

d o 



in 

Ov 



ON Ov 

c5 d 



c5 



d 



Ov f- 

o c5 



o 



vp 

CO CO 



o\ r~ 
■n vq, 

CO 



CO 



oo 

CO 



On 

m 

CO 



8 £ 

CO CN 



CS CO 



ON 

m 

CO 



H VO 

cn r- 
co cn 



cn 

00 



r-i r- 
t oo 



CO CO 



CO o\ 
m 

CN CO 



CN 

m 



CM 



Z 

o 

z 3 

Ed 

s 



S 

o 

U 

o 
z 

M 

z 

s 

CO 

M 



o 
c 
o 
u 

E 
o 



JS 

<J 

u 

8. 



•8 

% 

O 

o 



60 

c 

*S 
w 
u 

E 

n 

(9 



3 

X) 



> 

s 

o 

3 

o* 

U 



J3 
60 



u 
o. 

X 

u 



a 

u 

3 
O* 

O 



00 



-o 
u 



■5 o 

. CO 

^ § - 

~" e 

u 

_a 

c 

o 



3 

ca 



u 
o 
c 
u 

c 
u 

T3 

e 

3 

a 

•B 
o 
u 

>e 

u 
*3. 
S 

8 
js 

"H. 
E 

'Co 

•o 
c 

at 



c 
o 



3 



S 

O 
u 

X 

u 
"o, 

E 

o 
u 



c 

u 

s 

u 

c 

CM 



V3 
4) 

19 

X 

u 



3 

o 

u 



T3 

■8 

c 

u 

« 

u 

T3 
E 



ca 
u 

o 



J3 •- 



c 

JB 
W 

u 

I 

e 
o 
c 



u o u u 



E 

•o 
e 

CM 



(0 

E 

•o 
e 

as 



o 

'Jo 

E 

•o 
e 

CM 



u u 

C C 

3 3 



Si 

c 

o 
u 

*« 
E 



•S 
'to 

E 



•o 

u 

t-l 

w 

c 
u 

XI 
c 

s 



c 

u 



u 



u 
E 

c 

u 

o 
a. 

at 

c 



03 
U 

n 

'Jo 
E 



c 

O 

c 



c 
o 
"E 
'5. 
o 



o 
.<a 



-O 

u 



c 

u 



o. 

u 
ca 

*-» 

c 
u 
£ 



u 
c 



-a 

u 

ta 
X 



60 

.S 
'E 
o 

ca 
w 



u 

60 

ca 



c 

u 

o 



J3 



J3 
C 

o 

c 

_2 

S. 
E 

3 



a: D 



u u u 

T3 T3 T3 

c c c 

D D D 



u 
a 



c 

u 

T3 



(J 
U 



c 
u 



c 

u 



c 
o 



ca 

3 



o — 
n 

u 



g - 
c S3 

8 '-a 



•2 g « 

* S 



o 

S 

o 

> 

J 3 
ta 

ca 



u 



'S "5 q S3 '5 "5 8 1 



W3 

u 

00 

c 

ca 

JS 
X 

u 

u 
a 
o 



o 

o -s 

3 

o 

T3 
S 

ca 

to 



o 
u 
c 
o 



u 
c 

T3 
C 
3 

o 

w 
60 

u 

C8 
X) 



u 
o 
a 

C8 



T3 
C 

ca 

to 



o 

ca eg 

t; CB 



o 

5 .2 



CO 

> 



c 

o 



c 

o 



c 
o 
w 

T3 

c 

ca 

w 



U 

a 
o 
o 

T3 

S 
h* 

o 
a 



o 
u 



T3 
C 

ca 

w 



U 

0i 3 



u u u 

-O -O T3 

ecu 

D D 3 



in vo 



^" » * s d s s s* si s c* ss ss s s a 




e 

o 
u 

I 



2 
o 
+■» 

as 
l* 
+■* 

.52 o 

a <*> 

E " 
-o ^ 

<u 
*-» 

83 
*■* 

V) 



J* 
u ii 



Q 

CO 



u 

s 



Q 

CO 



u 

s 



Q 

CO 



9 

u 

s 



SI 8 

o © 



C) 5 



C— On 

© o 



§8 $ 

© i-l 



9 



oo 



n© 



S S 



3 3 

©° © © © © 



£ £ 3 $ a 

t*i ft t*i t*% fi 



S f8 £ Si 3 

o o o o o 



ci ci ci ci 



o r- 



S<T) tr> o 
fN fN *-< 
Ci Ci O O 



oo 



3 



o 



ci 



© 



ON ^ H 

t- oo oo 
© © o 



00 



oo 



O rt i-C 

oo oo o\ 
© © ©' 



P» <S Q >£> <* On On 
On 00 On On 0» F; >0 

N N N N « fi N 



N© N© O l*J «-J 00 00 
On On 00 [~ 00 55 00 



3 a s 

Ci (O CI CI o 



© O £; £2 
On r-t o 



$ §8 

© © 



<N i-l 

00 00 



00 OO 



5 



On 
© 



K-! P» <S 
O 00 
M « N N 



On 



oo 

o d 



On in 

5- vr> 



oo 



ON 



00 
© © 



00 



oo 



On 



s 

o 



a 
o 
« 

Z 

o 

m 

CO 

Z 
W 

s 

a. 
S 
O 

u 

o 
z 

z 

s 

CO 



8. 



u 
s 
D 



§ 

'S 

> 

■a 

a 

o 

•a 

u 

Cm 

o 

u 
a 

1 



i 1 



2 
o 

S 

u 

Q 



z 
o 

CO 

Z 
W 
B 



a. 
S 
O 

U 

O 

z 
3 



a 

© 

1 



B 

■5 



s 
o 



a 
EP 



a 
a 



o 

w 

c 
3 

■8 

s 
3 



u 

E 

u 



E 



B 

2 

O 

£ 



u 
•a 

60 
B 



s a 



•3 



U 



(3 
U 



(8 

£ 
£ 

3 



2 'S <« 



a. 



B 

u 

13 



Si 



B 
u 
•a 



B 

u 

13 



T3 



U 



(9 
U 
T3 

00 
B 

O 

£ 

to 



w .S- 



B 
O 

JS ^ 

o. u 



u 

JS 



g & «&■ «&■ <^ 

o 
u 
Q 



On 



c 
u 

13 



a 

o 



60 
B 

■5 
n 



B 
O 



0) 

£f 

o 

•o 
s 
a 

B 

4» 

B 

O 
u 

V 
JS 

■o 

B 

2 

e 
■8 

B 

3 



B 

O 



60 
B 

■5 

CO 

t 

a 

o 
«■» 
a 

V 

a 

o 
u 

•o 

.Si 

S 

JS 

•o 

B 

2 

e 

t3 
B 
3 



T3 
U 



O 
U 



ca 

JS 

£ 
o 



"o. 
E 



ca 

JS 



3 
60 
B 



B 

U 

B 

O 
u 

•o 

w 



u 

JS 

E 

o 



u 
B 

u 



u 
ca 



Q 2 



■a 

u 



u 
-a 



u 

JS 



B 

60 

O 

U 

02 



B 

O 



u 

B 
B 

ca 

£ 

w 

O 



u 

JS 



CN 



B 

60 

O 

U 

02 



B 

O 
C 

"5. 
o 

S 
o 



3 

60 



a 

o 



60 
B 



o 
o. 

3 

o. 



o 

JS 



JS 



u 
a 



u 
E 
o 

u 

w 

3 

O 



u 
u 

B 

u 

3 
O" 

u 



3 

o 



60 

c 

94 
JS 

"O 

a» 



60 

94 



B 

_a 

J3 
JS 

* 

>» 

i 



Q cu 



e 
S 

e 

V 
B 

3 



C9 

u 
a. 

w 

O 

B 



B 

U 



.8 



8 



o 
u 
OS 



On 



O 
B 
JX 

w 

a 

ca 
u 



£ 

3 

ca 

O 
JS 

3 



o 

u J= 



js "O ca 

JS 



B 

U 
13 



9 



9 

ERIC 



u 
o 



.8 



Q 



e 

CS 
U 

2 



CO o\ 
oo r-. 
o o o 



oo 



8 



CO <s 
cs ro 



co 



S 3 

o d o 



CO 
</"> 



CS CS 00 

» v| ^ 

fi 



m cs 
o o o 



00 

o 



lO 

in 



s 

o 



CS 
On 

o 



r- 



3 8 



rococococococs'cocococo 



r- 



1-1 

o 
2 

a 11 

•O 

< Z 

CO 
-*-<• 
kV5 



Q 

c/j 



u 

2 



O CO 

»-< ON 

CO CS 



§8 00 S 
O O O O 



co CO 



vo 

o o 



?2 £ 
o 



cs 

co co co 



d o 



o cs 

VD t-; 
ci CO 



.-I O 

o d 



cs 

00 



in 



cs 

vO 

o 



CS 

o 



CO 



oo 



CO 



d 



s 

CO 



co oo £ 
r~ v© oo 

odd 



on 

CO CO 



s 



CM 
CO 



la 

g » 



u 

2 



oo §8 
d o 



oo 



5 S 



CS CS CO cO 



§ S rC 

d d o 



m q <-< 
oo * t 

CO CO CO 



cs 
o 

CO 



lo S vo >0 06 i*; 

d d d d d d d 



«-l CS 

c- oo 

CO CO 



H CS 
CO CO 



CS 
CO 



co cs p- 
odd 



5; S 

CO CO CO 



cs m 

NO ~< 
CO CO 



u 

c 

o 
u 

1 -8 

5 u 



2 .a 

z .5 

u a 

S 3 



I 

.8 



u 
OS 




in 
9 



a 

08 
S 

o 



(A 

a 

S J 

.8 1 

c u 

— o 



"8 : 

o 

e © 

w ^ 

> (A 

3 3 

s a 

v) eg 

u S 

•c o 



cu 
c 

cs 
<—« 

c 



3 

•2 

X) 



u 
u 

'I 



a> 
"S. 

'3 



.t; a. J3 y fe 



.© hi U I- 



u 
> 



w H w 

« J 

3) 3) S> 

u u u 

hi V" V" 

.b .b .b 



u 
,cj 
<—-i 

i_ 

s 

u 

J3 



2 cu 

IS u 

cs J3 

X) -* 

P a 



a 

o 



cs 
CU 

a 

o 

W9 
U 

hi 

CU 



•a o 



CS 
CU 



c 

CS 



T3 

e 

CS 



T3 
C 

CS 



V) 

a 

o 

a 



S3 S3 S 



3 

u 



4 



3 3 . 
60 SO 



— y cs 

2 * « 

B u u 

U C hi 

j3 u u 

w hi IA 



8 

c 
o 



* i i s i i 



« I 

cs CS 

CU u 

. *" 



CS CS 



s 



3 



5? 



cs 



u 
u 

ha 
CU 

u 

J3 



CO 
in 



O 
«) 
e 
o 

'5 

o 

hi 

u 

55 



o 
o 

•o 
c 

CS 

u 

hi 

3 
.3 



s- 

u 
u 



u 



c 

cs 



3> 



c 



3> 

u 



u 
o 

*s 

> 

u 
> 



CS 
CU 

u 

JS 



ON 

m 



9 

ERIC 



S2 
o 

■§s 

u « 



Q 

00 



o o 



ts 3 

co <s 



CS VO 
*© r- 

d o 



<n 
co 



3 5? 

co co 



in 



3 



to oo 
to to 



m 
co 



CM 

r- 



in .-h 

oo 

co to 



2 3* 



(S 
VO 
to 



•0 
to 



co 
VO 
to 



38 in oo s6 0\ 
o o o o o" o 



s> S g * 



O tS Tf CO «-i 
«* On to O 
CO CO tS ro ts ts 



CO 

u 

o 
2 

.S3 o 
2 «*> 

E » 

V5 



Q 

00 



(N r- 

VO ON 
O © 



On »— " 
O C] 

ci en 



On co m on on c— 

vo r» is vq >n m oo 
d d d d d d d 



m c~» <s ts co co m 
<n >n vq c~; oq oo in 

CO CO ro to 



r- co 
r» vo 
d d 



r- 
>n 
co 



r- 
co 



r- 
ro 
co 



co ts r~ co m 

on f» oo m on on 
d d d d d d 



co co" 



m 
in 



r- 
co 
ts" 



vO 
r- 
ts 



w II 

tS II 



Q 



u 

2 



tS 8 

to to 



s 



<n On 

r~ in 



to co 



3 

co 



f- 
to 



8 



On 



ts r- op vo 
in oq oo on 

CO CO CO CO 



VD tS Tt 

vo m 
odd 



On 



fS ~* 

in r- 



m 
<n 



3 $ 2 

r*". c*") c*") 



m m 
to «-< 

CO ts" 



m 

ON 



r- 
to 
ts 



in 



a 



a 

3 
O 

a 

o 



a. k. 



o 
o 



8 vo 



c 

3 
O 

c 

o 



T3 

c 

co 

I 

w 

o 
u 

(0 



u 

I 

T3 

a 

to 



ca 



E 

"3 

u 

-a 
c 

(4 



1 £ .2 e « s *. 

2 « — b c 5 S 



tfl ri 

O 3 

S JF 

o o 

T3 O 



& 

3 

W5 

•o 

a 
ca 

1/3 

u 
> 



u 

'•a" 
co 

e 

CO 

a 

3 
O 

e 



c 

3 
O 

c 

O 



c 

CO 



> 



s 

3 

o 
c 



> 

'5 

u 
co 



« <2 E 



co 
00 

o 



u 
s 



co 
Cu 

E 

o 



_3 
Q, 



c 
u 

E 

u 
u 
w 

esc 
co 



co 



o 
o. 



VO 



o 
E 



co 
u 
'5 
u 

C 

-o 
c 

co 

u 

'5 
is 
u 



u 
c 



B 
O 



u 

e- 

u 
o 

co 

E 
g 



u 
-a 



o 



u 



*V3 



i 



a 

u 

E 
j> 

"3 

u 

x> 



U5 



U 
T3 
h* 
O 

•a 

ha 

-o 

B 
co 

U5 



OA 

ca u u 

0C B B 

b 3 p 



U 

oo 

CO 
3 
00 

s 

CO 



E 

o 



CO 



u 
-o 



B 

o 



B 

a 

•o 

B 
co 

US 



3 

O 
u 

'•5 
"to 
o 
o 



B 

u 
E 

JO 

"o 
u 

"!o 

CO 

X> 

•o 

B 

CO 
W3 



o u 

-o -o 

B B 

3 3 



CO 

vo 



3 ■<> 



vO 



VO 



s 



On 

>o 



ts 



co 



■<i- in 



9 

ERIC 



C/J 

2 Q 

tu 

£z s 

CO 



on 

o o 



8 8 



Cn eN r -1 ?2 
rr) vo r-_ in 

dodo 



i-i i-i 
r- r- r». 
do© 



oo 
o 



On 

o 



IT) 

c- 



On 
O 



00 



oo 



i-i Q 
oo r~ C> 

d d o 



o 

u. 
*■* 

i/i o 

.5 **> 
E " 

< Z 

4) 
*■* 
A 
w 

CO 



Q 

C/3 



a 
u 



o\ no 
d o 



NO 



8 
d 



d o 



r-> O t~- 

o\ On no 
r«S ci 



00 



i-i 

d o 



3 R 
<*5 tr> 



d 



cn 
d 



d 



oo 
d 



8 
d 



<*5 



d 



d 



oo 



« ii 

32 



Q 

C/3 



s 



a §8 

d d 



00 r-t 

ri <*5 



ir> 

d o 



no. 



c5 



IT) 



On 
no 



<*5 



S S S S 3 S S S 

dddooooo 



NO 

d 



NO 



8 8 



<*5 



ON 



s s 



oo NO 
d d o 



NO 



m cn <n 
^ m oq 
t«S cS t«S 



1 

1-4 

u 



u 
ca 

8. 



I 

c 

« 

c 
o 
a 

a 



T5 -8 



I 
a 



a 

.2 
<2 
'u 

a 

3 

c 

o 



H a. 

M 

a 

o 

E 

u 

Q 



a 

u 

T3 



z 
a. 

CM 



u 

=3 
U 



E 

w 
•3 

a 

ea 

a 

o a 

•JS .2 

ea 
a 
o 



3 
o 

•a 

3 

o 

B 

U 



ca 

8. 

C/3 



o 

ea 

a, 
•0 



ca 

Q. 

« 

ca 

B 



1 
O 

U 



l-H fN 

00 00 



S — 

V 1 



u 

.52 
*C 

D. 

O 

u 

a. 

C9 



ca 



C/3 



>■■) 
oo 



5- 

ca 
i- 

3 

ca 

1 
ca 
E 

E 

ca 

w 

oo 

^3 



5 a 5 



ca 



oo 



B 
3 

E 

E 

o 
u 

c 

'3 



u 
i— * 

ca 



•5 



13 -s 



ca .a g .a 

&, D. D 



'o. 
o 



60 
B 

•c 

A) 



U 

ca 

s 
8 

■o 

B 

ca 



B t- 
4> ca 



E 

<2 

s 

o 

c 
o 



oo 



S3 

Z J 



E 

«i2 

B 

O 

i/i 

c 
o 

'c 

'o. 
o 



u 

B 

ca 



b <a 

u ca 
Q Z 



oo 



B 

O 



ca 

3 



™ T3 
J3 P 



oo 
ca 

3 

eo 

B 

ca 



J2 
~£ 

x> 
ca 
u 

B 

(X 

ca 

< 



B 

.2 
ca 

B 

ca 



u 
00 
ca 

u 

s 

B 

u 



£3 

u 
.*2 
a 
•3 
E 



8- 8. •£ 



u 

O nm 



u 

Jet 
CO 

2L 



= O 



s 

o 



B 

u 

3 

c 

-o 

B 

ca 

B 
o 

'5 

u 
c 

C 

a 

« 
s 
o 



ca 
u 



"S. 
a. 

3 

00 



On ON Sn 



U 
3 
O* 

U 

OS 



On 



ca 

E 
S 

3 

C/3 



On 



5 

•O 
3 

,o 
"3 
•o 

ea 

w 
0C 



On 



B 
U 

3 



a 
o 



■3 I I S -s i 



o 

u 



u 
•a 



d 

ERIC 



o 

I 



Q 



c 
<« 
u 



r~ <s co 
irj r-~ r~ 
odd 



r- 



9 



8 



ro 



On 



00 
U-1 



IT) >Ti 



CI 
cO 



2° "2 



rs t~ o 
v-i «-> in 

www 



9 S 

o o 



IT) *H 

c5 c5 



O 

*•» 

CO 

u 
*•» 

'£« 
S 11 

-O >7 
< Z 

u 

*•» 
CO 
w 

</5 



Cs 

to' 



a 

u 



9 

© 



fS 3 ^ S ? n 
d c5 o o c5 c5 o 



r- r- t; o 
>© cs no ^ 

m m m m 



S 5 



r- 
oq 

co co co 



5? 



</-> 
On 

O O O 



s 

o 



00 

o 



o 



Sco c-> 
oo <s 
« N « N n « 



8 8 S 



in 



oo o 
On v~> 
o d 



r- r- © 
v© <s r-; 
w m w 



co 



o 

V5 



r- r- 
oo 8 s ; 
c5 c> 



2 & 



Q 

CO 



e 



00 

co 



8 £ 



d d d o o o o 



o? 7? 
d 



§ f) On t~ S 
co co cm r-i co co 



«s c- 



CO 00 <N 

oo oo on 
d d o 



oo 
on 
d 



On 

d 



CO 

no 



oo © 
d d 



SggS^SScSlo 
coHcMrsrscor<Sr<Scn 



© 
r- 



o\ 



u 
3 

•a 



4> — O 

a 2 o. 



as u 
u 

.S 
o 

•§> - 



3 

en 

EP 



s 
o 
a 
u 

1 



e 8. -s 



I * 

u 

on 



(9 

a 

o 



a 

(9 



U 
t—t 

.2 
o 

u. 
Q. 
Q. 

(9 

3 



u 
o 
a 
u 
-a 
a 



s 

'§ 

Ji 
"a. 

E 



o 
a 

u 

(9 



u 
a 

(9 

a 

"« 
a 

J3 

(9 

"S a 
« o 

i « 

a a 



a 
o 

JS 



a 

> 
u 

a 

C9 

<u 
X) 



^ Q 



a 

O 



O. 



8! S o 



o 



CO 

o 



C9 

.s 
r 



S 



a 

u 

T3 

a 



a 
o 



a 



a 

o 



U5 • — 

o 

o 



.a « o 
a 3 

a 

u 

J3 
U 



E 
o 

u 

•? 

o 



o 



2 

CO 

ec 

"35 
o 
o. 

E 

o 
U 



E 

e 
o 

a 

o 



55 S: JS 



■o 
a 

(9 
*^ 

a 

u 



8 



u 

■? 

o 

(A 

a 

"5 



S3 2 3 



w Q* *^ • — 



oo 
a 



(9 
O. 

E 

o 

o 

T3 

a 

(9 

U 
*^ 
(9 
*^ 

U 

02 



o 



cu 
o 



E 

JS 

e 
o 

a 

o 



a. 
o 

a 

C9 
T3 

a 

u 
Q 



00 

o 



>. 

3 

o 



a 

o 

CO 

E 

1m 

a 

T3 

a 

C9 

Kfi 

at 
U 

■o 

I 
(9 

a 
< 



eg 

E 

60 

a 



o 

Q. 

a. 



_o 
"53 
> 
o 

T3 
T3 

a 

(9 
U 

u 
a 



s 
-a 

_o 
a 

C9 



> 



(9 

•o 



-a <o 
1 £ 



a. 
o 



o 
_u 

(9 
> 



•S u as 5 

U W *T2 ^ 



.2 

3 

s 

O 
> 

60 

a 



o 

(9 

U 
*^ 
(9 



a 
o 

E 

u 

Q 



8 
a 
u 
"■5 
a 

(9 

U 
J3 



a 

(9 



O 
w 
U 
w 
O 

E 



19 



U 
w 

.2 

O. 
O 
w 
O. 
D. 
(9 

U 

8P 



a 



o 
n 

U-i — 

.2 3 
- en 

•£ _c 

60 *2 
.a > 
s t> 

a 

OS 
OA 

a 



u 
u 
a 

T3 
O 



J5 2 S 

3 1 3 
u 



a 

(9 
D. 

CO 

a 



u 
_o 
'S3 

(9 
X) 
4) 

J< 

(9 

E 



w U 
U J* 

S3 a. 

<9 

E 
E 

C9 
00 

o 

es 
X) 



B 
00 

8 

u 

as 



B 

a 

» 

B 

C9 



B 

3 




to 
£- 

O 

Is 

u 11 

jg z 

u 



q 

c/5 



c 

u 

2 



00 t-; 
O O O 



f- 5 E: 
o o\ f\ 
e-i 



SO O 

o o 



00 00 

o o 



<s 2; 
f-i r'i 

<s <s 



in 

00 

o 



00 

o 



v£5 00 



,-H 1-H 00 

oo oo 06 
o o o 



r- 

O 



o 8 



t> 



r- 
c- 



a 



tN Ov 



oi s; 

00 00 

d d 



to 
t- 

O 
•«-» 
co 
u 

4-* 

«3 O 

c ° 

"O -7 

< Z 

V 
•«-» 
CO 
•«-» 

oo 



Q 

c/5 



c 

to 
u 

s 



s 



t— to 
odd 



o r- 



9 



§ 58 

d o 



8 



«-h tN 
r- oo 
d d 



8 

oi 



oi 



d 



oi 



CO 

d 



oo 



N rt N 



d 



s 

oi 



g ffi 88 P R 9 ?S 

o o d d d o o 



t- o o g £ £; 
n m m fi « ^ 



op oi 
d d 



2 £ 



ii 



u 
co 



a 

to 
u 



js £ 8 

odd 



2 S S 

<ri ei «<S 



d 



d o 



8 



S S 

oi 



3; 

d 



in 
o> 



ID 

o 



oi oi 



3 §8 

d d 



oi 
o 



in 
o> 
d 



OI 



On 
00 



d 



oi oi 



00 



oo 

ON 

oi 



d 



oi 



oo 



o o 



in g 

00 

d d 



CO 




3 

o 



o 
'to 1 



B 

.to 



s 
o 

S 
u 

Q 



3 



C 

o 
u 



•5 "SE 



3 



w tO 

_«j .2 

c. to 

e c/3 i: 



3 
o 
c 

oo 
'■5 
c 



3 

o 



o 

.2 'ST 
^ B 



oo -S 

^ 3 
J2 



B 
.to 



u — 

E o 

a ^ 



c 
o 
u 

u 



a; 



3 
00 



o 
a. 
B 
u 

c 

o 
o 



3 
00 



c 

CO 

u 



5> 



c 

to 
o 



c 

00 



o 
u 

I-. 

to 
_y" 

3 

B 



o 

to 

IE 
c/i 

o 



s 

to 
a. 
oo 



3 

00 



c 

CO 



o 
a. 

£? 

C3 
w 

8. 

B 

u 

C 

o 
u 



2 ^ iS" T5 -2 



o 

3 

XI 



c 
o 



8 1 
o 



to 



3 

.8 

CO 

o 



c 

00 



B 

tfc" 

c 
o 
-e 



a R S 3 



u 
c 
u 
u 

CO 
CO 



- 3 3 S -5» '= S 3 



c 
o 

B 
u 
Q 



<N 




a 

CO 

o. 

c« 

00 

c 

•a 

« 



u 

& 
o 



I 

~5 
u 



3 

u 

"w 

o 

o 

JS 

_c 

'c 
to 
o. 

tyj 

tM 

o 
_o 
o 

u 

u 

w 

c 

'3 



9 

ERIC 



CO 



2 

h 

u » 



V) 



a 

10 
U 



SI 8! § S 8 I 8 « S !? S P 5 So ? S R S ? 

CJO'-'OOOOOOOOOOOOOOOOOO 



n ? 8 8 s » 

N N N N N N N 



00 



pS 



pi 
o 



pi 
oo 



5? 
pS 



5? 



o 

pS 



in 



s 

pS 



GO 



t- 

CO 



SO 

O 

2 

co o 

c <*> 

S « 
•a -7 

u 

« 



a 



c 

to 
u 



© © 



00 



oo 



3 



o 



s 

© 



PI 
pS 



51 3 

pS cn 



rH i-< f~ 
O VO VO 
<-i O O 



00 



5 5 

rn pS 



oo 



pi 
pS 



p-> 
pS 



© 



pi 
00 
pS 



oo 
d 



p-> 



Ov 

d 



o 
pS 



CO 

pS 



pi 
d 



IT) 

ro 



oo 
pS 



5 5? ?R 

odd 



§ 8 So 
pS pS Pi 



w ii 



c 

to 
O 



oo «-i 
©' 



3 S! S S 8 ^ § M 51 

i-idddddddddd 



P"> 
v£> 



8 8 

rn pS 



c4 



o 



p-> 



■li- 
pS 



s 

pi 



pS 



.-I 

p- vo 
pi pS 



oo 



p- 
oo 



p-> 



in 



in 

00 

d 



o 
pS 



R 



pS 



d 



in 
pS 



pi 
p- 



2 $ 
pS pS 



a. 
o 



3 



OO 
_c 

J3 
U 

to 



u 

e 

o 



u 
> 



CM 



a <3 <3 



3 



60 

a 

IS 
o 
to 
u 



eo 

e 

o 



u 
> 



o 
u 

60 

c 



to 
u 



o 
•o 
o 

JS 

u 

s 

■w 

c 

u 

3 
o 



60 
C 

c 

to 



O u 

O C 

S 3 

E § 
u 

J3 to 

1 I 

l2 Q 



c 

o 



c 

to 

60 



J3 



to 
"to 

> 



c 

o 



3 



to « 

^ c 

o u 

•» 3 

c to 

t0 r . . 

r o 

g. Q. 

■ 3 
O 



u 



u 
c 
l> 

w 

to 
to 



c 

o 

E 

o 

Q 



60 



c 

u 

c 

o 
o 

o 

N 



to 

l> 



c 
o 

c 

o 
o 



60 

c 



B 

U 

3 



o 
U 



60 

u 

"5 

o 
c 



c 



o 
to 
E 



S .§ E 



•o 



3 

o 



* ° 

u 

to 
3 

o 



o 

u 
to 
3 

o 

13 



60 
"3 

c 

1) 



10 

c 

o 



a. 
o 
t~ 

D. 

a. 

u 
to 
E 



Q. 

c 

to 
U 

to 

E 

•o 
c 

to 



to 
E 



to 

_3 

"to 

> 



to 

_3 

"to 

> 



to 

_3 

"to 

> 



'5b 
u 



60 
C 

IS 
u 
to 
u 



u 
to 

-o 
c 
to 



u 
to 



60 

u 



60 

c 

IS 
o 
to 
u 



N >, w w w w 
3 vE. CO to 10 to 



CO 
_3 

"to 

> 

UJ 



o. 
to 

o 
>» 



to 
u 

to 
_3^ 

o 
_u 
"o 

00 



c 

o 



** B — 2 



CO 

a. 

1/5 



c 

u 

(A 



CO 

u 



3 

o 



g 2 «, a u a 



o. 
o 



60 

c 

o 

E 

CO 



3 

o 



c 

to 



a .x2 



.2 

a. 

o 

Q. 

a. 
to 



E 

o 
oo 



E 
CO 

a. 

00 

a 

CO 
b« 

60 



B 

O 

-o 

3 



D 



o 

"Bb 
u 



•a i 

u 5 

[2 .3 



H 1- tS «8 



j3 u >- ^ 

•S !2 to 



E 



S 

o 



o. r - S 

^2 3 to 



to 

3 

"to 

> 

u 



to 

_3 

to 
> 



60 
_E 

*E 
O 



U 
3 

o 



a. 

o 



E 

o 
U 



Q. 

a. 



a. 
_o 

o 
> 

Q 



3 

O 



.3 « 



B 

•o 

3 



- C w « 



O 
60 
CO 
60 
E 

w 



=i o w •= 



CO 



60 
E 



PI 



5 



5 



o 



p^ 
1/1 



p" 



0C 

in 



o 

ERIC 



to 

y 

u II 

u 
to 



C/J 



c 
CO 
u 

2 



o o o o 



in 
On 

o 



Ov 
oo 



« 8 

o o 



^ ^8 s s 

N N N 



O* 

c4 



00 iC 
m on o\ 

odd 



s 



0\ 
oo 



o 
«■* 

to 

<A O 

*c «*» 
S " 

< Z 

V5 



Q 



n 

to 
u 



d 



<n >o 



Sin 
P 

i-I i-i o 



r-- r-. 
o 



s 

d 



o 

o <-< o 



5n°£88SS°£8 

m m to r<i m m 



r- 
oo 



to " 

2z 



Q 



n 

s 



oo r-^ 
d o 



o 



s ^ § 

<r> <ri to 



in 



d 



00 



c4 



ON 



ON 

d 



ON 



ON 



m 
oo 



.8 



s 

60 

u 
> 



oo 
s 

•a 

3 

I 

•O 

3 

O 
J3 



e 

o 



e 

o 



o 
E 



■a 

3 

o 

■g O 



oo 
co 
e 
co 



8 
u 
e 
u 

u 

CO 

e 

co 



06 £ 



e 

o 

E 
u 
Q 



s 
co 
a. 



oo 
s 

IE 

u 

03 

u 



u 

•5b 
u 

CO U 
— -<= 
.. *^ 



CO 
J5 



O 

T3 
3 

CO 



U 
8 

3 



oo 

8 



CO 



3 

u 
"3 

3 



(J 



CO 



o 

CO 

t/) w w 



u 

8 

3 



H S3 



o E 3 



o 

8 
U 
u 

I 

8 

(0 



v) 
8 
O 

a 

u 
Q 



3 

CO 

oo 



M o .a 



CO 

8 

O 



«5 .S .3 



15 
o 



8 
u 

8 
U 
u 

I 

CO 

8 

OS 

u 



8 

o 
E 
u 
Q 



u 

8 



U 



3 

oo 
& 

8 

U 

'5 

13 
O 

a 

u . 
oo 

CO 

3> 

8 

CO 



3 

u 

-O 
8 

CO 



u 

8 

3 



u 

« 
"E 

CO 

a. 

>% 
T3 
3 



CO 



8 

•8 

3 

o 



s 

8 

U 

S u 

.8 § 

u "<3 

00 t; 

" 3 

3 ^ 

00 3 

8 U 



J3 

.3 
•ob 

8 



u 



2 S 



VO 



m 
vO 



5 



^5 



o 



o 

ERIC 



Appendix G 

Mean Importance Ratin gs: Subgroups of Respondents 
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